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Abstract 

This  thesis  examines  the  impact  of  acquisition  reform  initiatives  implemented 
since  1993  on  contract  cost  performance.  Many  initiatives  implemented  during  the  1990s 
focused  on  saving  the  government  money  in  procuring  weapon  systems.  For  decades, 
cost  overruns  have  plagued  Department  of  Defense  weapons  systems  development  and 
procurement  costing  the  government  money.  The  passage  of  the  Federal  Acquisition 
Streamlining  Act  (FAS A)  of  1994  and  the  Clinger-Cohen  Act  of  1996,  marked 
significant  congressional  action  on  aiding  the  Department  of  Defense  develop  and 
procure  systems  cheaper.  Conclusions  drawn  in  this  thesis  may  aid  leadership  in 
supporting  current  initiatives  and  drafting  new  changes. 

This  research  effort  studied  204  contracts  completed  between  January  1,  1994  and 
December  31,  2001.  It  was  discovered  that  cost  performance  for  contracts  completed 
after  reform  initiative  implementation  was  no  different  than  cost  performance  on 
contracts  completed  before  implementation.  The  research  also  analyzed  cost 
performance  on  all  active  contracts  from  1970  to  2002  against  the  backdrop  of 
acquisition  reform  studies  and  commissions  over  the  same  time  period,  to  examine  any 
trends  or  time  lags  between  reform  implementation  and  contract  cost  performance 
change.  The  results  indicate  some  evidence  of  cost  performance  change  following  the 
different  studies  and  commissions.  Possible  explanations  and  implications  of  these 
results,  and  suggestions  for  future  research  are  provided. 
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AN  ANALYSIS  OF  THE  IMPLEMENTATION  OF  ACQUISITION  REFORM 


INITIATIVES  AND  CONTRACT  COST  VARIANCE 


I.  Introduction 


We  must  do  more  with  less. 

-  The  list  is  long 


General  Issue 

The  decade  of  the  1990s  was  a  period  of  change  for  the  United  States  military, 
both  operationally  and  budgetary.  The  end  of  the  Cold  War  and  the  success  of  Operation 
Desert  Storm  resulted  in  increased  operations  tempo,  a  personnel  reduction  of  34%,  and  a 
budget  reduction  of  40%  (Cordesman,  1999:  2-3;  CBO,  2000:  1-2).  The  “Do  More  With 
Less”  adage  was  bom  and  became  a  common  statement  among  Department  of  Defense 
(DoD)  leadership. 

The  Clinton  Administration’s  first  term  faced  these  issues  head  on.  The  military’s 
worldwide  tasking  and  deployments  increased  as  overseas  bases  were  closed  and 
remaining  bases  faced  aging  facilities  and  infrastructure  in  need  of  repair,  all  requiring 
additional  funds  (CBO,  2000:  1-2).  At  the  same  time,  the  DoD  continued  to  develop  and 
procure  new  weapon  systems  and  modifications  to  existing  weapon  systems,  which  were 
more  technologically  advanced  and  more  expensive  than  legacy  systems  (Goure,  2000: 

1).  To  make  the  federal  budget  pieces  fit,  funding  for  national  defense  decreased  in  order 


1 


to  keep  day-to-day  operations  running  (see  Figure  1).  In  fact,  procurement  spending 
declined  by  59%  between  fiscal  years  1987  to  1997  (CRS  Report,  2001:  CRS-2). 


In  order  to  continue  some  level  of  modernization,  the  acquisition  and  procurement 
process  within  the  federal  government,  and  more  specifically,  within  the  DoD  required 
change.  Acquisition  Reform,  as  we  know  it  today,  was  reintroduced  in  an  effort  to 
procure  weapon  systems  better,  faster  and  cheaper  (CRS  Report,  2001:  CRS-2). 

Specific  Issue 

The  defense  acquisition  system  experienced  “reform”  movements  in  the  1970s 
and  the  1980s.  Despite  reform  focused  on  reducing  costs,  research  shows  that  weapon 
system  programs  continued  to  overrun  cost  and  schedule  constraints  (Searle,  1997).  In 
fact,  a  1993  RAND  study  indicated  that  cost  growth  of  DoD  programs  since  the  mid 
1960s  fluctuated  around  20  percent  (Drezner  and  others,  1993:20).  The  last  three  DoD 
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Acquisition  chiefs  lauded  the  current  Acquisition  Reform  movement  successful. 

However,  some  are  not  so  optimistic  and  pleased  with  the  results.  In  April  2000,  before 
the  Senate  Armed  Services  Committee  Readiness  Subcommittee,  Senator  Carl  Levin 
stated  that  many  contracts  have  failed  to  use  or  have  overused  the  increased  flexibility 
provided  by  Acquisition  Reform  and  therefore,  have  not  realized  cost  reductions.  Current 
research  on  this  subject  is  optimistic  and  indicates  that  contract  cost  performance  woes 
are  on  the  rebound  and  that  Acquisition  Reform  initiatives  are  in  fact  reducing  the  cost  of 
buying  new  weapon  systems  (Christensen  and  Templin,  2002). 

Scope  and  Limitations  of  the  Study 

This  study  focuses  on  the  timeline  of  current  reform  initiative  implementation, 
with  an  emphasis  on  cost  reduction- focused  initiatives.  This  study  assesses  if  DoD 
weapon  system  contract  cost  performance  is  improving  or  not  and  how  any  cost 
performance  trends  (positive  or  negative)  relate  to  the  implementation  timeline.  The 
Defense  Acquisition  Executive  Summary  (DAES)  database  will  be  analyzed  to  perform 
the  program  cost  performance  assessment. 

There  are  two  limitations  in  this  study  important  to  note.  First,  based  on  prior  and 
current  research,  the  analysis  assumes  that  Acquisition  Reform  initiatives  prior  to  the 
Clinton  Administration  had  no  positive  effect  on  the  contract  cost  performance  and 
therefore,  any  changes  in  performance  are  a  result  of  the  current  reform  movement. 
Second,  because  only  seven  years  have  passed  since  the  current  Acquisition  Reform 
movement  started,  the  full  impact  and  success  cannot  be  assessed  in  this  study. 
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Research  Objectives 

This  research  study  has  two  main  objectives.  First,  to  determine  if  a  mapping 
between  cost  initiatives  and  cost  growth  improvement  exists  and  if  so,  what  is  the  time 
period  between  implementation  of  an  initiative  and  the  results.  Based  on  this 
relationship,  the  focus  will  be  on  developing  a  model  or  “rule  of  thumb”  to  estimate  or 
forecast  an  impact  window  or  time  frame  in  which  to  expect  results  for  future  cost-related 
Acquisition  Reform  policies.  The  end  goals  are  to  provide  some  evidence  to  DoD 
leadership  that  Acquisition  Reform  is  positively  affecting  contract  cost  performance  and 
to  provide  evidence  that  it  takes  time  for  such  initiatives  to  bear  fruit.  Successful 
accomplishment  of  these  goals  would  allow  DoD  to  use  this  evidence  in  the  future  when 
implementing,  supporting  and  justifying  new  changes  to  the  Acquisition  process  and 
assessing  associated  results. 

In  order  to  achieve  these  objectives,  the  analysis  is  broken  into  phases.  Phase  One 
will  analyze  and  compare  contract  cost  performance  for  pre-  and  post-reform  periods  to 
see  if  there  are  indications  that  acquisition  reform  initiatives  are  having  an  impact,  either 
positive  or  negative.  Phase  Two  will  look  at  contract  cost  performance  over  time  to  see  if 
there  is  a  “rule  of  thumb”  relationship  between  acquisition  reform  initiatives  and  cost 
performance  improvement.  Dependent  upon  the  results  of  Phase  One  and  Phase  Two,  a 
third  phase  of  analysis  will  be  conducted  with  different  pre-  and  post-reform  periods  used 
to  see  if  there  are  indications  of  cost  performance  impact. 
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Organization  of  the  Study 

This  chapter  established  the  motivation  for  analyzing  the  topic  and  the  research 
objectives.  Chapter  2  describes  past  and  present  Acquisition  Reform  efforts,  describes 
Earned  Value  Management  and  provides  details  on  past  research  in  this  subject  area. 
Chapter  3  details  the  methodology  used  to  analyze  the  cost  performance  and  rule  of 
thumb  development  and  Chapter  4  provides  the  results  of  the  analysis.  Finally,  Chapter  5 
provides  conclusions  from  the  study  and  recommendations  for  further  research. 
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II.  Literature  Review 


DoD  has  achieved  a  large  measure  of  success  with  Acquisition  Reform. 

-  Honorable  Paul  G.  Kaminski,  USD(A&T) 

26  February  1997 

Real  reform  in  our  acquisition  of  weapons  and  major  systems  has  taken  place  in 
recent  years. 

-  Honorable  Jacques  S.  Gansler,  USD(A&T) 

Testimony  before  Subcommittee  on  Acquisition  and  Technology  of  the 
Senate  Armed  Sendees  Committee,  18  March  1998 

These  reform  initiatives  are  so  obligatory  with  each  change  at  the  helm  that  they  can 
bring  a  “same  package,  different  name”  quality  with  them.  It’s  a  shame-but  not  a 
surprise -that  so  much  cynicism  has  grown  around  such  a  worthy  word  as  reform. 

-  Honorable  E.  C.  “Pete  ”  Aldridge,  USD(AT&L) 

30  April  2002 


Chapter  Overview 

This  chapter  provides  a  brief  overview,  historical  and  current,  of  Acquisition 
Reform  efforts,  a  description  of  the  Earned  Value  Management  system  utilized  to  assess 
Department  of  Defense  (DoD)  program  cost  performance,  and  a  review  of  past  research 
conducted  in  analyzing  Acquisition  Reform  and  DoD  contract  cost  performance. 


Acquisition  Reform 

The  historical  overview  starts  with  Deputy  Secretary  of  Defense  David  Packard’s 
changes  in  the  1970s  and  concludes  with  the  1989  Defense  Management  Report.  The 
current  reform  overview  begins  with  the  National  Performance  Review  (NPR)  and 
concludes  with  a  focus  on  initiatives  that  affect  cost  performance  and  cost  control. 
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Historical  Acquisition  Reform  Overview  (1970’s  -  1992) 

Table  1  lists  several  defense  acquisition  studies  conducted  from  the  1960s  through 
the  1980s.  This  historical  reform  overview  will  focus  on  those  highlighted,  as  these  are 
the  major  acquisition  studies  orchestrated  by  the  Executive  Branch. 


Table  1.  Significant  Defense  Acquisition  Studies  (Jones,  1996:405) 


_ Report  by _ Issued 

Peck  &  Scherer  (Harvard  Business  School)  1962, 1964 

Blue  Ribbon  Defense  Panel  (Fitzhugh  Commission)  1970 

Commission  on  Government  Procurement  1972 

J.  Ronald  Fox  (Harvard  Business  School)  1974 

Military  Services  and  Secretary  of  Defense  1974-75 

Defense  Science  Board  Summer  Study  (Acquisition  Cycle  Task  Force)  1977 

Defense  Resources  Board  1977 

DoD  Resource  Management  Study  1979 

Jacques  S.  Gansler  1980 

Acquisition  Improvement  Task  Force  (Carlucci  Initiatives)  1981 

Special  Panel  on  Defense  Procurement  Procedures  1982 

Grace  Commission  1983 

Special  Task  Force  on  Selected  Defense  Procurement  Matters  1984 

Georgetown  Center  for  Strategic  and  International  Studies  1985 

Blue  Ribbon  Commission  on  Defense  Management  (Packard  Commission)  1986 

Defense  Management  Review _ 1989 


Packard 


The  Vietnam  War  and  rising  weapon  system  procurement  costs  created  a  Defense 
Department  spending  draw  down  (Jones,  1999:  402).  Secretary  of  Defense  Melvin  Laird 
and  his  Deputy,  David  Packard,  realized  the  need  for  better  control  over  the  acquisition 
process  (Ferrara,  1996:110).  In  1969,  Secretary  Packard  returned  the  authority  for  the 
acquisition  process  to  the  military  services  and  created  the  Defense  Systems  Acquisition 
Review  Council  (DSARC)  to  act  as  the  advisory  group  to  the  Defense  Secretary  (Ferrara, 
1996:111;  Jones,  1999:402).  With  the  creation  of  the  DSARC,  Secretary  Packard 
established  milestones  and  phases  to  monitor  and  assess  program  status  and  to  streamline 
the  acquisition  process  (Jones,  1999:403). 
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In  May  1970,  Secretary  Packard  issued  a  memorandum  listing  broad  themes  such 

as  decentralized  execution,  streamlined  management  structures  and  use  of  appropriate 

contract  mechanisms  (Ferrara,  1996:111).  This  memorandum  would  become  the 

foundation  for  DoD  Directive  (DoDD)  5000.1,  completed  in  July  1971  (Ferrara, 

1996:111).  Secretary  Packard’s  vision  was  the  following: 

Successful  development,  production,  and  deployment  of  major  defense  systems 
are  primarily  dependent  upon  competent  people,  rational  priorities,  and  clearly 
defined  responsibilities.  Responsibility  and  authority  for  the  acquisition  of  major 
defense  systems  shall  be  decentralized  to  the  maximum  practicable  extent 
consistent  with  the  urgency  and  importance  of  each  program. 

The  development  and  production  of  a  major  defense  system  shall  be  managed  by 
a  single  individual  (program  manager)  who  shall  have  a  charter  that  provides 
sufficient  authority  to  accomplish  recognized  program  objectives.  Layers  of 
authority  between  program  manager  and  his  Component  Head  shall  be 
minimum. .  .[the]  assignment  and  tenure  of  program  managers  shall  be  a  matter  of 
concern  to  DoD  Component  Heads  and  shall  reflect  career  incentives  designed  to 
attract,  retain,  and  reward  competent  personnel  (Ferrara,  1996: 1 1 1). 

The  final  section  of  DoDD  5000. 1  contained  the  following  guidance: 

•  Wherever  feasible,  operational  needs  shall  be  satisfied  through  the  use  of 
existing  military  or  commercial  hardware; 

•  Practical  tradeoffs  shall  be  made  between  system  capability,  cost,  and 
schedule; 

•  Logistic  support  shall  be  considered  as  a  principal  design  parameter; 

•  Schedules  shall  be  structured  to  avoid  unnecessary  overlapping  or 
concurrency; 

•  Test  and  evaluation  shall  commence  as  early  as  possible; 

•  Contract  type  shall  be  consistent  with  all  program  characteristics,  including 
risk; 

•  Source  selection  decisions  shall  take  into  account  the  contractor’s  capability; 

•  Develop  a  necessary  defense  system  on  a  timely  and  cost-effective  basis;  and 
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•  Documentation  shall  be  generated  in  the  minimum  amount  to  satisfy 
necessary  and  specific  management  needs. 

Packard’s  early  ideology  established  acquisition  reforms  that  DoD  still  operates  under 
today.  Packard’s  vision  would  be  reflected  in  the  studies  that  followed  over  the  next  30 
years. 

Carlucci  Initiatives 

The  Acquisition  Improvement  Task  Force,  also  known  as  the  Defense  Acquisition 
Improvement  Program  or  the  Carlucci  Initiatives,  was  directed  by  Secretary  of  Defense 
Casper  Weinberger  in  1981  to  look  at  all  aspects  of  defense  acquisition  (Ferrara, 
1996:118;  Jones,  1999:406).  The  “Reagan  buildup”  of  the  1980s  required  reductions  in 
weapon  system  costs  and  development  durations  (Jones,  1999:406).  Led  by  Deputy 
Secretary  of  Defense  Frank  Carlucci,  the  task  force  resulted  in  32  initiatives  (Table  2)  to 
improve  the  acquisition  process.  However,  due  to  a  lack  of  acceptance  from  Congress 
and  industry,  coupled  with  rumors  of  fraud,  waste  and  abuse,  most  of  the  32  initiatives 
were  never  implemented  (Gregory,  1989:60).  Initiatives  that  survived  addressed  multi¬ 
year  procurement  and  fixed- price  contracts,  greater  competition,  stabilized  programs  and 
realistic  budgeting  (Jones,  1999:406).  These  initiatives  made  their  way  into  the 
acquisition  process  with  the  1982  revision  of  DoDD  5000.1,  as  evidenced  by  this  excerpt 
from  the  1982  version: 

Improved  readiness  and  sustainability  are  primary  objectives  of  the  acquisition 
process....  Reasonable  stability  in  acquisition  programs  is  necessary  to  carry  out 
effective,  efficient,  and  timely  acquisitions.  To  achieve  stability,  DoD 
Components  shall  conduct  effective  evolutionary  alternatives,  estimate  and 
budget  realistically,  [and]  plan  to  achieve  economical  rates  of  production  (Ferrara, 
1996:119). 
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Table  2.  Carlucci  Initiatives  (Adams,  1984, 15) 


1.  Reaffirm  Acquisition  Management  Principles  17. 

2.  Increase  Use  of  Preplanned  Product  18. 

Improvement  19. 

3.  Implement  Multiyear  Procurement  20. 

4.  Increase  Program  Stability  21. 

5.  Encourage  Capital  Investment  to  Enhance 

Productivity  22. 

6.  Budget  to  Most  Likely  Costs  23. 

7.  Use  Economical  Production  Rates  24. 

8.  Assure  Appropriate  Contract  Type  25. 

9.  Improve  System  Support  and  Readiness  26. 

1 0.  Reduce  Administrative  Costs  and  Time  27. 

1 1 .  Budget  for  Technological  Risk 

12.  Provide  Front-end  Funding  for  Test  Hardware  28. 

13.  Reduce  Governmental  Legislation  Related  29. 

Acquisition  30. 

1 4.  Reduce  number  of  DoD  Directives  31 . 

15.  Enhance  Funding  Flexibility  32. 

16.  Provide  Contractor  Incentives  to  Improve  33. 


Decrease  DSARC  Briefing  and  Data  Requirements 
Budget  for  Inflation 
Forecast  Business  Base  Conditions 
Improve  Source  Selection  Process 
Develop  and  Use  Standard  Operation  and 
Support  Systems 

Provide  More  Appropriate  Design-to-Cost  Goals 
Implement  Acquisition  Process  Decisions 
Reduce  DSARC  Milestones 
Submit  MENS  (later  JMSNS)  with  Service  POM 
Revise  DSARC  Membership 
Retain  USDR&E  as  Defense  Acquisition 
Executive 

Raise  Dollar  Threshold  for  DSARC  Review 
Integrate  DSARC  and  PPBS  Process 
Increase  PM  Visibility  of  Support  Resources 
Improve  Reliability  and  Support 
Increase  Competition 
Enhance  the  Defense  Industrial  Base 
(added  1984) _ 


The  General  Accounting  Office  (GAO)  completed  a  final  report  analyzing  the 
Carlucci  Initiatives  in  July  1986  (Munechika,  1997:8).  The  GAO  concluded,  “top  level 
commitment  to  change  did  not  filter  down  to  the  program  manager  level”  and  that  the 
“commitment  to  the  improvement  program  ha[d]  dissipated”  (GAO,  1986:2;  Munechika, 
1997:8).  In  fact,  after  only  five  years  implementing  the  Carlucci  Initiatives,  the  DoD  was 
moving  forward  with  the  plan  of  monitoring  results  of  the  initiatives,  resulting  in  the  lack 
of  success  (Munechika,  1997:9). 

Grace  Commission 

On  June  30,  1982,  President  Reagan  established  the  President’s  Private  Sector 
Survey  on  Cost  Control  (PPSSCC)  under  the  guidance  of  J.  Peter  Grace,  to  identify 
opportunities  for  better  efficiency  and  reduced  costs  that  could  result  from  any  form  of 
executive  or  legislative  action  (US  Congress,  1984:2).  This  survey  group  was  recognized 
as  the  Grace  Commission.  The  Grace  Commission  consisted  of  161  chief  executive 
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officers  of  major  corporations  and  private  sector  experts,  and  totaled  more  than  2,000 


total  participants  (US  Congress,  1984:3). 

The  Grace  Commission  submitted  2,478  recommendations  to  the  President  on 
reforms  for  the  government  (US  Congress,  1984:24).  Only  112  of  the  2,478  pertained  to 
DoD  operations,  and  21  of  the  112  involved  changes  to  the  acquisition  process  (US 
Congress,  1984).  The  major  recommendations  are  listed  below  in  Table  3. 


Table  3.  Grace  Commission  Recommendations  (House  Armed  Services  Committee, 

1985:3) 

1 .  Greater  use  of  multiyear  contracting  to  improve  program  stability 

2.  Prioritize  all  weapons  systems 

3.  Streamline  and  strengthen  the  contract  selection  process 

4.  Upgrade  cost  estimating 

5.  Enhance  the  role,  responsibility,  authority  and  accountability  of  the  PM 

6.  Increase  the  use  of  dual  sources,  throughout  the  life  of  the  program 

7.  Increase  emphasis  on  the  Spare  Parts  Breakout  Program  to  identify  and  obtain  spare  parts 
from  sources  other  than  the  Prime  Contractor 

8.  Consolidate  responsibility  for  contract  administration  activity  at  the  level  of  OSD 

9.  Simplify/streamline  the  30,000  pages  of  regulation  related  to  Defense  procurement 

10.  Mandate  use  of  common  components,  subsystems  and  equipment  by  all  services 

11.  Eliminate  the  use  of  unnecessary  military  specifications 

12.  Outsource  commercial  functions 

13.  Incentivize  government  employees _ 


The  only  shortfall  with  the  Grace  Commission  was  that  the  recommendations 
came  shortly  after  the  Carlucci  Initiatives  were  published.  The  DoD  believed  they 
addressed  the  Commission  recommendations  by  instituting  the  Carlucci  Initiatives 
(Munechika,  1997:12).  Therefore,  the  Grace  Commission  recommendations  for 
acquisition  reform  eventually  faded  away  much  like  many  of  the  Carlucci  Initiatives 
(Munechika,  1997:12). 
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Packard  Commission 


On  July  15,  1985,  responding  to  continued  stories  of  acquisition  problems  and 
contractor  overcharging,  President  Reagan  initiated  the  Blue  Ribbon  Commission  on 
Defense  Management,  chaired  by  former  Deputy  Defense  Secretary  David  Packard 
(Munechika,  1997:12;  Jones,  1999:407).  The  Packard  Commission  focused  on  broad 
organizational  changes  and  would  become  the  Reagan  Administration’s  most  recognized 
acquisition  reform  movement  (Jones,  1999:407).  The  Commission’s  charter  was  to 
review  and  make  recommendations  in  the  following  areas: 

•  Adequacy  of  the  defense  acquisition  process,  industrial  base  and  current  law 
and  regulations; 

•  Adequacy  of  current  authority  and  control  of  Secretary  of  Defense ; 

•  Responsibilities  of  the  Joint  Chiefs  of  Staff; 

•  Adequacy  of  the  Unified  and  Specified  Command  system  in  providing 
effective  planning  and  use  of  forces; 

•  Value  and  continued  role  of  intervening  layers  of  command  on  military 
forces ; 

•  Procedures  for  developing  and  fielding  military  systems  incorporating  new 
technologies  in  a  timely  fashion; 

•  Congressional  oversight  and  investigative  procedures  relating  to  the  DoD; 

•  Recommend  how  to  improve  the  effectiveness  and  stability  of  resources 
allocation  for  defense  (Blue  Ribbon  Commission,  1996:27-28). 

The  Commission  examined  how  to  make  the  military  acquisition  system  more  efficient 

like  commercial  organizations  (Munechika,  1997:13).  By  June  1986,  less  than  one  year 

from  organizing,  the  Commission  submitted  the  final  report.  Table  4  shows  the  Packard 

Commission’s  Formula  for  Action  to  make  the  process  more  efficient. 
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Table  4.  Packard  Commission’s  Formula  for  Action 


A.  Streamline  Acquisition  Organization  and  Procedures 

1 .  Create  new  Under  Secretary  of  Defense  for  Acqusition  position 

2.  Each  service  should  establish  a  comparable  Service  Acquisition  Executive  (SAE) 

3.  Each  SAE  should  appoint  Program  Executive  Officers  (PEO) 

4.  Program  managers  report  directly  to  PEOs 

5.  Substantially  reduce  the  number  of  acquisition  personnel 

6.  Recodify  federal  laws  into  a  single,  greatly  simplified  statute 

B.  Use  Technology  to  Reduce  Cost 

1 .  Emphasize  building  and  testing  prototypes  to  demonstrate  new  technology 

2.  Operational  testing  should  begin  early  in  development 

3.  Prototypes  can  provide  a  basis  for  improved  cost  estimating 

C.  Balance  Cost  and  Performance 

1 .  Restructure  Joint  Requirements  and  Management  Board  leadership 

2.  Joint  Requirements  Management  Board  should  define  weapon  requirements  and  provide 
tradeoff  between  cost  and  performance 

D.  Stabilize  Programs 

1.  Baseline  programs  and  use  multi-year  funding 

E.  Expand  the  Use  of  Commercial  Products 

1 .  Do  not  rely  on  military  specifications 

2.  Use  off-the-shelf  products  as  much  as  possible 

F.  Increase  the  Use  of  Competition 

1 .  Focus  on  more  effective  competition,  modeled  on  commercial  practices 

2.  Emphasize  quality  and  past  performance  as  well  as  price 

G.  Enhance  the  Quality  of  Acquisition  Personnel 

1 .  Allow  Secretary  of  Defense  to  establish  flexible  personnel  management  practices 

2.  Recommend  new  personnel  management  system  for  acquisition  personnel, 

contracting  officers  and  scientists  and  engineers 


On  April  2,  1986,  President  Reagan  signed  a  directive  to  implement  the  Packard 
Commission’s  recommendations  (Blue  Ribbon  Commission,  1996:33).  The  Packard 
Commission  laid  the  groundwork  for  the  passing  of  both  the  Goldwater-Nichols  Act  and 
the  Defense  Acquisition  Improvement  Act  of  1986;  both  contained  sweeping  changes  to 
the  acquisition  process  and  overall  DoD  management  (Munechika,  1997:15).  Also,  the 
applicable  Packard  Commission  recommendations  were  written  into  a  new  version  of  the 
DoDD  5000.1;  specifically  the  new  chain  of  authority  (Ferrara,  1996:120). 

By  1990,  the  progress  reports  on  the  Packard  Commission  recommendations  were 
positive.  Particular  emphasis  had  been  placed  on  program  stability  and  cost  control 
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(Munechika,  1997: 15).  The  same  report  noted  that  only  the  enhancement  of  the  DoD 
acquisition  personnel  portion  of  the  Formula  for  Action  was  not  a  success  story 
(Munechika,  1997:15).  The  Bush  Administration  would  take  an  aggressive  look  at  this 
“success”  upon  taking  office  in  1989. 

Defense  Management  Report 

Upon  taking  office  in  1989,  President  Bush  requested  a  National  Security  Review 
with  the  goal  of  developing  a  plan  to  fully  implement  the  Packard  Commission 
recommendations,  further  improve  the  defense  acquisition  system,  and  better  manage  the 
DoD  (Cheney,  1989:i).  The  Defense  Management  Report  (DMR)  was  completed  in  July 
1989.  The  DMR  took  an  analytical  view  of  the  DoD  based  on  the  Packard  Commission 
recommendations  and  assessed  how  far  along  the  department  was  in  implementation  and 
most  importantly,  what  remained  to  be  accomplished.  Secretary  of  Defense  Cheney 
noted  that  to  realize  President  Bush’s  objectives  for  improving  the  acquisition  process, 
the  following  would  need  to  occur: 

•  Teamwork  among  DoD’s  senior  managers; 

•  Sound,  longer-range  planning  and  better  means  for  managing  available 
resources; 

•  More  discipline  in  what  weapon  systems  we  buy  and  how  we  buy  them; 

•  Better  management  of  the  people  we  rely  on  to  produce  such  systems; 

•  An  environment  that  promotes  steady  progress  in  cutting  costs  and  increasing 
quality  and  productivity;  and 

•  Adherence  to  the  highest  ethical  standards  (Cheney,  1989:27). 

Notable  improvements  included  chartering  the  Joint  Requirements  Oversight 

Council  (JROC)  to  enhance  the  requirements  generation  and  weapon  system  validation; 
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establishing  an  Acquisition  Corps  and  improving  professional  education  and  training; 
strengthening  the  role  of  the  Under  Secretary  of  Defense  for  Acquisition;  and 
consolidating  the  contract  administration  functions  under  a  single  organization  -  the 
Defense  Logistics  Agency  (Munechika,  1997:16;  Jones,  1999:404).  The  DMR  concluded 
with  a  vitally  important  point;  actions  will  take  time,  effort,  consensus  and  commitment 
(Cheney,  1989:27). 

By  the  end  of  1991,  major  steps  in  implementing  many  of  Secretary  Cheney’s 
recommendations  had  occurred.  The  Defense  Contract  Management  Command  (now 
Agency)  was  created  under  the  Defense  Logistics  Agency,  consisting  of  a  reduction  to 
five  district  offices  and  manpower  reductions  in  all  offices  (Munechika,  1997:16).  To 
address  the  discipline  issues,  the  DoDD  5000  series  was  expanded  significantly.  The 
new  series  contained  clear  guidelines  with  an  attempt  to  streamline  the  regulation  regime 
by  consolidating  all  previous  directives,  instructions  and  policy  memoranda  (Ferrara, 
1996:121;  Munechika,  1997:17).  As  a  result,  the  new  series  contained  over  900  pages 
and  was  more  than  15  times  larger  than  any  previous  version. 

Summary 

This  section  provided  a  brief  overview  of  some  of  the  major  acquisition  reform 
initiatives  between  1970  and  1992.  From  a  historical  point  of  view,  it  is  evident  that 
while  the  reform  studies  and  commissions  have  changed  names,  many  of  the  ideas  and 
recommendations  remained  the  same  between  Packard  (1970),  Carlucci  (1981),  Grace 
(1983),  Packard  (1986)  and  the  Defense  Management  Report  (1989). 
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Current  Acquisition  Reform  Overview  (1993  -  Today) 

This  section  defines  the  current  acquisition  reform  period  as  the  reform  initiatives 
that  began  with  the  Clinton  Administration,  specifically  the  National  Performance 
Review,  led  by  Vice  President  Gore.  This  section  focuses  on  the  National  Performance 
Review  (NPR)  conclusions,  the  resulting  legislative  changes,  and  concludes  with  a 
synopsis  of  reform  initiatives  that  have  had  a  direct  focus  on  cost  control. 

National  Performance  Review 

The  National  Performance  Review  (NPR)  was  a  similar  review  to  what  the  Grace 
Commission  accomplished  10  years  prior,  in  that  it  performed  a  detailed  look  at  all 
government  activities  to  find  areas  for  improvement.  The  first  report  was  released  on 
7  September  1993  (Munechika,  1997:31).  This  report  contained  384  recommendations 
that  spanned  all  facets  of  government  operations,  including  federal  procurement 
(Munechika,  1997:31).  The  NPR  pointed  out  that  in  the  DoD,  there  were  at  least  889 
laws  affecting  procurement,  the  Federal  Acquisition  Regulation  (FAR)  was  over  1,600 
pages,  and  was  further  supplemented  with  over  2,900  pages  of  agency- specific 
procurement  regulations  (Reinventing  Federal  Procurement,  1993:5).  Table  5  gives  a 
brief  layout  of  the  recommendations  for  procurement  reform  that  came  out  of  the  NPR. 

Analyzing  the  list,  it  is  easy  to  discern  similarities  between  prior  commission 
recommendations  and  issues  identified  by  the  NPR.  In  1995,  the  NPR  reconvened  and 
accomplished  a  second  assessment  to  look  for  any  governmental  functions  that  could  be 
terminated,  privatized  or  restructured  (National  Partnership  for  Reinventing  Government, 
1999).  The  actual  savings  suggested  by  the  NPR  are  contested  between  what  the  Clinton 
Administration  estimated  versus  what  the  General  Accounting  Office  (GAO)  and  other 
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private  groups  could  estimate.  For  the  purposes  of  this  report,  the  true  savings  are  not 
important.  The  importance  of  the  NPR  is  that  it  reinvigorated  the  Acquisition  Reform 
movement  in  the  DoD  (Munechika,  1997:31). 


Table  5.  National  Performance  Review  Procurement  Reform  Recommendations 
(Reinventing  Federal  Procurement,  1993) 

PROCOI  Reframe  Acquisition  Policy 

-  Reduce  rules  and  regulations 
PROC02  Build  an  Innovative  Procurement  Workforce 

-  Better  education  and  training 
PROC03  Encourage  More  Procurement  Innovation 

-  Test  new  methods  with  pilot  programs 

PROC04  Establish  New  Simplified  Acquisition  Threshold  and  Procedures 

-  Low  cost  procedures  for  small  purchases 
PROC06  Amend  Protest  Rules 

-  Increased  communication  between  buyers  and  sellers 
PROC08  Reform  Information  Technology  Procurement 

-  Decrease  time  to  purchase  computer  equipment 
PROC09  Lower  Costs  and  Reduce  Bureaucracy  in  Small  Purchases 

Through  the  Use  of  Purchase  Cards 

-  IMPAC  Card  use 

PROC13  Foster  Reliance  on  the  Commercial  Marketplace 

-  Reduce  reliance  on  government-specific  specifications 
PROC1 5  Encourage  Best  Value  Procurement 

-  Lowest  bidder  is  not  always  best! 

PROC16  Promote  Excellence  in  Vendor  Performance 

-  Use  past  performance  in  contract  award  decisions 
PROC18  Authorize  multiyear  contracts 

PROC1 9  Conform  Certain  Statutory  Requirements  for  Civilian  Agencies 
to  Those  of  Defense  Agencies 

-  Raise  minimums  for  cost  and  pricing  data  certification 


Federal  Acquisition  Streamlining  Act  (FAS A)  of  1994 
The  FAS  A  of  1994  incorporated  much  of  Vice  President  Gore’s  NPR 
recommendations  for  creating  a  “government  that  works  better  and  costs  less”  (FASA 
DSMC,  2002).  The  intent  was  to  overhaul  the  cumbersome  and  complex  procurement 
system  of  the  federal  government  and,  more  importantly,  the  DoD  (FASA  DSMC,  2002). 
Table  6  highlights  some  of  the  changes  included  in  the  FASA  of  1994. 
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Table  6.  Federal  Acquisition  Streamlining  Act  of  1994  (FASA  DSMC,  2002) 

-  Eliminated  paperwork  and  record  keeping  requirements 

-  Allowed  direct  micropurchases 

-  Exempted  commercial  purchases  from  cost  and  pricing  data 

and  cost  accounting  standards 

-  Reserved  small  dollar  acquisitions  for  small  businesses 

-  Improved  bid  protest  and  contract  administration  procedures 

-  Required  evaluation  of  past  performance  before  contract  award 

-  Raised  Truth  in  Negotiation  Act  threshold  for  required  cost  and 

pricing  data _ 

The  result  of  the  new  law  caused  a  ripple  in  the  federal  and  defense  regulations 
related  to  acquisition  and  program  management.  In  fact,  the  new  reform  push  started 
before  passage  of  the  law.  In  June  1993,  Colleen  Preston  was  appointed  as  the  Deputy 
Under  Secretary  of  Defense  for  Acquisition  Reform  [DUSD(AR)]  (Munechika,  1997:31). 
In  1995,  Under  Secretary  of  Defense  for  Acquisition  and  Technology  (USD[AT&L]), 
Paul  Kaminski  called  for  the  revision  of  the  DoD  5000  series  to  incorporate  the  new 
flexibility  and  streamlined  practices  created  by  the  FASA  of  1994  (DoD  Directive  5000 
Series  rewrite  DSMC,  2002).  In  fact,  the  FASA  required  that  all  federal  regulation 
change  proposals  be  published  not  later  than  210  days  after  the  law  was  enacted 
(GAO/NSIAD-96- 139:1). 

Title  V  of  the  FASA  of  1994  contained  provisions  for  performance-based 
acquisition  management  designed  to  foster  the  development  of  1)  measurable  cost, 
schedule,  and  performance  goals,  and  2)  incentives  for  acquisition  personnel  to  reach 
these  goals  (GAO/NSIAD-97-22BR:!).  At  the  time  of  the  GAO’s  report,  the  DoD  had 
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complied  with  the  applicable  part  of  Title  V  (part  1)  by  Fiscal  Year  1995  (GAO/NSIAD- 
97-22BR:2). 

Clinger- Cohen  Act  of  1996 

In  1996,  Congress  enacted  the  Information  Technology  Management  Reform  Act 
and  the  Federal  Acquisition  Reform  Act  (Clinger- Cohen  DSMC,  2002).  These  two  acts 
together  are  better  known  today  as  the  Clinger-Cohen  Act.  A  quote  by  Under  Secretary 
of  Defense  for  Acquisition  and  Technology,  Paul  Kaminski,  summarizes  what  the  act 
accomplished: 

The  Clinger-Cohen  Act  of  1996  (formerly  known  as  the  Federal 
Acquisition  Reform  Act  of  1996  (FARA)  and  the  Information  Technology 
Management  Reform  Act  of  1996  (ITMRA))  further  advance  the  changes 
made  by  FASA.  The  Clinger-Cohen  Act  provides  a  number  of  significant 
opportunities  for  DoD  to  further  streamline  and  reduce  non- value  added 
steps  in  the  acquisition  process.  Among  the  most  significant  changes 
authorized  by  the  Act  is  a  test  of  the  use  of  the  Simplified  Acquisition 
Procedures  (SAP)  for  commercial  items  between  the  simplified  acquisition 
threshold  of  $100,000  and  $5  million.  This  should  allow  DoD  to  reduce 
its  administrative  costs  and  overhead  costs  for  DoD’s  vendor  base  for 
purchases  of  relatively  low  risk  items.  This  change  eliminated 
government-  unique  requirements  previously  cited  by  industry  as  a  barrier 
to  doing  business  with  DoD.  The  Act  also  provides  the  authority  for 
contracting  activities  to  use  SAPs  for  all  requirements  between  $50,000 
and  the  SAP  while  the  government  works  to  fully  implement  Electronic 
Commerce/Electronic  Data  Interchange  (EC/EDI).  (Clinger-Cohen 
DSMC,  2002) 

Important  to  the  analysis  of  cost  growth,  and  more  specifically  cost  variance,  is 
that  the  Act  provided  some  relief  from  cumbersome  and  non-  value  added  processes  that 
increased  costs  of  programs  (Clinger-Cohen  DSMC,  2002).  Additionally,  the  Clinger- 
Cohen  Act  established  statutory  standards  for  acquisition  personnel  and  required  each 
federal  agency  involved  in  procurement  to  budget  funds  specifically  for  education  and 
training  (AFMC  Acquisition  Reform  Guide,  1999:5).  This  was  a  key  step  forward. 
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Personnel  would  not  be  able  to  utilize  the  new  flexibility  and  concepts  unless  they  were 
trained  and  educated  first. 

Summary 

Current  period  reform  initiatives  are  not  conceptually  different  from  ideas 
presented  in  the  previous  section.  The  major  difference  was  the  direct  involvement  of 
Congress  in  mandating  changes  in  procedures  and  regulations  to  provide  increased 
flexibility.  The  enactment  of  the  FAS  A  of  1994  implemented  much  of  the  NPR 
recommendations.  The  enactment  of  Clinger-Cohen  of  1996  (FAS A  II)  further  enhanced 
the  groundwork  established  two  years  earlier.  The  next  section  provides  additional  detail 
for  specific  changes  implemented  by  the  DoD  to  take  advantage  of  the  new  flexibility  and 
attempt  to  make  headway  into  controlling,  and  ultimately  reducing,  the  cost  of  defense 
procurement. 
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Review  of  Implemented  Initiatives 

Integrated  Product  and  Process  Development 

Integrated  Product  and  Process  Development  (IPPD)  is  a  concept  used  by  private 
industry  since  the  early  1980s  and  now  “borrowed”  by  the  DoD  (DoD  Guide  to  IPPD, 
1996:1-2).  At  the  core  of  IPPD  implementation  is  the  use  of  Integrated  Product  Teams 
(IPTs)  (DoD  Guide  to  IPPD,  1996:ix).  Figure  3  provides  a  graphical  representation  of 
the  IPPD  process. 


Figure  2.  Generic  IPPD  Iterative  Process  (DoD  Guide  to  IPPD,  1996). 


IPPD  is  defined  as  the  following: 

A  management  process  that  integrates  all  activities  from  product  concept 
through  production/field  support,  using  a  multi-functional  team  to 
simultaneously  optimize  the  product  and  its  manufacturing  and 
sustainment  processes  to  meet  cost  and  performance  objectives.  (DoD 
Guide  to  IPPD,  1996:1-3) 

Proper  implementation  of  the  IPPD  process  requires  an  understanding  of  the  key 
tenets.  The  tenets  are  self-explanatory  from  their  titles  and  are  listed  in  Table  7. 
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Table  7.  Key  Tenets  of  IPPD  (DoD  Guide  to  IPPD,  1996:1-7  -  1-8) 

1.  Customer  Focus 

2.  Concurrent  Development  of  Products  and  Processes 

3.  Early  and  Continuous  Life  Cycle  Planning 

4.  Maximize  Flexibility  for  Optimization  and  Use  of  Contractor  Approaches 

5.  Encourage  Robust  Design  and  Improved  Process  Capability 

6.  Event-Driven  Scheduling 

7.  Multidisciplinary  Teamwork 

8.  Empowerment 

9.  Seamless  Management  Tools 

10.  Proactive  Identification  and  Management  of  Risk _ 


A  28  April  1995  memorandum  from  the  Under  Secretary  of  Defense  for 
Acquisition,  Technology  and  Logistics,  Paul  Kaminski  and  a  10  May  1995  memorandum 
from  the  Secretary  of  Defense  William  Perry  officially  initiated  the  DoD’s  push  for  IPPD 
development  and  use  (Ferrara  and  Johnson,  1995:4). 

The  DoD  leadership  objectives  for  IPT  are  contained  in  Table  8.  Under  Secretary 
of  Defense  Kaminski  pointed  out  that  the  use  of  teams  “allows  for  early-on  and 
continuous  insight  by  all  stakeholders  in  a  program”  (FAR  Rewrite  DSMC,  2002). 


Table  8.  DoD  IPT  Objectives  (Ferrara  and  Johnson,  1995:2) 

-  Creation  of  an  acqusition  system  that  capitalizes  on  the  strengths  of  all  participants 

in  the  acquisition  process  to  develop  programs  with  the  highest  opportunity  for  success. 

-  Fostering  the  early  and  active  participation  of  OSD  and  Component  staff  organizations  with 

program  office  teams  to  develop  a  sound  and  executable  acquisition  strategy,  and  identify 
and  resolve  issues  as  they  arise,  not  during  the  final  decision  meeting. 

-  Transforming  historically  adversarial  relationships,  especially  between  headquarters  staff 

organizations  and  program  office  teams,  into  productive  partnerships. 

-  Placing  renewed  emphasis  on  the  importance  of  working  as  cross-functional  teams  to 

maximize  overall  performance. _ 


In  1993,  Wagner  and  White  investigated  the  F-22  System  Program  Office  (SPO) 
to  see  how  they  were  utilizing  IPPD  and  IPTs.  Lessons  learned  consisted  of  the 
realization  that  properly  organized  and  implemented  teams  were  effective  and  that  each 
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organization  must  tailor  their  teams  to  the  respective  organization  or  product/processes 
involved  (Wagner  and  White,  1993:5-5).  The  researchers  also  noted  that  communication 
was  key  to  making  the  process  flow  smoothly  (Wagner  and  White,  1993:5-6). 

Cost  as  an  Independent  Variable 

Cost  as  an  Independent  Variable  (CAIV)  is  the  DoD’s  acquisition  methodology 
for  making  technical  and  schedule  performance  a  function  of  available  budget  resources 
(Koreisha  and  LaPlaca-Mars,  1997).  This  initiative  drives  decision  makers  to  consider 
trading  performance  away  to  reduce  or  maintain  cost  (CRS  Report,  2001:CRS-6).  In 
December  1994,  the  Defense  Manufacturing  Council  (DMC),  headed  by  the  Principal 
Deputy  Under  Secretary  of  Defense  for  Acquisition  and  Technology,  met  to  formulate 
action  plans  to  implement  CAIV  into  the  acquisition  culture  (Focused  Approach, 
1995:49).  The  action  plan  for  CAIV  implementation  is  presented  in  Table  9. 

Table  9.  CAIV  Implementation  Action  Plan  (Focused  Approach,  1995:51) 

-  Develop  a  DoD  integrated  team  approach  (process)  to  establish  mission  needs,  tradeoffs 

and  cost  goals 

-  Establish  a  joint  Service  Acquisition  Executive  IPT  under  the  auspices  of  the  DMC 

-  Start  the  process  of  changing  "values"  (within  DoD) 

-  Make  unit  cost  objectives  and  cost/performance  tradeoffs  a  major  topic  of  each  Defense 

Acquisition  Board  meeting 

-  Establish  budget  and  resource  planning  and  analysis  procedures  compatible  with 

cost/performance  tradeoffs  as  a  main  acquisition  management  approach _ 

CAIV  strategy  entails  setting  aggressive,  realistic  cost  objectives  for  acquiring 
defense  systems,  and  managing  risks  to  obtain  those  objectives  (CAIV  Working  Group 
Report,  1995:1).  CAIV  is  made  possible  through  the  increased  flexibility  introduced  by 
both  the  FASA  of  1994  and  the  Clinger-Cohen  Act  of  1996.  A  key  element  to  making 
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CAIV  successful  is  proper  motivation  of  industry,  achieved  through  competition  and 
contract  incentives  (CAIV  Working  Group  Report,  1995:7).  Competition  includes 
making  life-cycle  costs  a  factor  in  source  selection  versus  the  previous  way  of  looking  at 
only  near  term  costs  (CAIV  Working  Group  Report,  1995:7).  When  competition 
between  multiple  vendors  is  not  possible,  contract  incentives  can  be  used  to  provide 
profit  motivation  for  contractors  to  reduce  costs  (CAIV  Working  Group  Report,  1995:7). 
The  first  edition  (published  in  1996)  of  DoDD  5000. 2R  to  include  CAIV  related  concepts 
required  each  acquisition  program  to  provide  incentives  to  contractors  to  meet  or  exceed 
cost  objectives  (Gaddis,  1998:35). 

Single  Process  Initiative 

In  December  1995,  Secretary  of  Defense  William  Perry  and  Under  Secretary  of 
Defense  for  Acquisition  and  Technology,  Paul  Kaminski,  introduced  the  Single  Process 
Initiative  (SPI)  (Kaminski,  1997:11).  The  intent  of  the  SPI  was  to  allow  contractors  to 
utilize  the  same  practices  and  procedures  while  performing  government  work  as  they 
would  while  performing  commercial  work.  The  requirement  to  segregate  commercial 
and  defense  related  efforts  for  similar  products  adds  costs  to  the  end  product  through 
increased  company  overhead  costs  (Perry,  1994:4).  In  other  words,  more  processes  for 
the  contractor  equates  to  more  cost  to  the  government. 

The  end  objectives  of  implementing  the  SPI  were:  1)  save  money,  2)  obtain  a 
better  product,  and  3)  foster  a  more  competitive  industry  (Kaminski,  1997:11).  SPI 
forced  contractors  to  consolidate  or  eliminate  multiple  processes,  specifications,  and 
standards  for  a  product  whether  it  was  for  commercial  or  defense  (CRS  Report, 
2001:CRS-6).  SPI  implementation  has  allowed  greater  civil/military  integration  and 
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paved  the  way  for  greater  emphasis  on  using  commercial  specifications  instead  of  rigid 


and  expensive  military  specifications  (Gansler,  1998).  The  early  success  of  SPI  is 
indicated  in  the  following  quote: 

One  of  the  benchmarks  of  success  in  our  acquisition  reform  effort  is  the 
speed  with  which  we  can  implement  specific  cost  saving  initiatives.  This 
is  one  of  the  major  contributions  of  the  Single  Process  Initiative.  It  has 
been  a  huge  success  in  paving  the  way  for  the  Department’s  conversion  to 
commercial  practices,  while  already  saving  almost  $500  million  in  costs. 

-  Honorable  Jacques  S.  Gansler,  USD(AT&L) 

29  September  1998,  SPI  Recognition  Awards  Ceremony 


Earned  Value  Management 

Earned  Value  Management  (EVM)  is  “a  tool  that  allows  both  government  and 
contractor  program  managers  to  have  visibility  into  technical,  cost,  and  schedule  progress 
on  their  contracts”  (DoD  EVM  Guide,  1997:2).  Beginning  with  the  Navy’s  Polaris 
program,  the  DoD  made  EVM  use  policy  in  1967  (Abba,  1995:1).  At  this  point,  the  tool 
was  known  as  Cost/Schedule  Control  Systems  Criteria  (C/SCSC).  The  35  criteria  were 
adopted  from  private  industry  best  practices  and  provided  a  contractor  with  minimum 
requirements  for  their  management  control  systems  (Abba,  1995:1).  During  the  1970s 
and  1980s  the  C/SCSC  continued  to  become  burdensome  and  became  viewed  as 
“compulsory  financial  reporting  requirements  as  opposed  to  valuable  and  fundamental 
management  tools”  (Abba,  1995:1). 

In  1989,  the  C/SCSC  policy  transferred  from  the  control  of  the  DoD  Comptroller 
to  the  control  of  the  Under  Secretary  of  Defense  for  Acquisition  (Abba,  1995: 1). 
However,  it  took  longer  for  DoD  managers  to  change  their  attitudes  regarding  the  “value” 
of  C/SCSC.  Following  “a  series  of  management  disasters”,  including  the  Navy’s  A- 12 
program,  DoD  began  to  view  EVM  as  a  vital  management  control  tool  (Abba,  1995:2). 
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The  early  to  mid-1990s  saw  many  refinements  to  the  requirements  and  to  the 
implementation  practices.  This  created  increased  acceptance  and  use  of  EVM  by 
government,  industry  and  foreign  countries  (Abba,  1995:2).  “Earned  value,  adapted  by 
DoD  nearly  30  years  ago  from  industrial  practices,  is  being  readapted  to  meet 
management  needs  in  today’s  high-tech  commercial  world”  (Abba,  1995:3). 

Concepts  and  Criteria 

Program  managers  and  other  senior  decision  makers  need  insight  into  how  their 
acquisition  programs  are  performing,  both  from  a  schedule  and  cost  perspective.  Use  of 
an  EVM  system  provides  managers  with  this  data  which: 

•  Relate  time-phased  budgets  to  specific  contract  tasks  and/or  statements  of 
work; 

•  Indicate  work  progress; 

•  Properly  relate  cost,  schedule  and  technical  accomplishment 

•  Are  valid,  timely  and  auditable; 

•  Supply  managers  with  information  at  a  practical  level  of  summarization;  and 

•  Are  derived  from  the  same  internal  earned  value  management  systems  used  by 
the  contractor  to  manage  the  contract  (DoD  EVM  Guide,  1997:2). 

The  main  premise  behind  use  of  the  EVM  tool  is  to  tailor  the  system  to  properly  “match” 

the  using  organization,  the  products,  and  working  relationships  involved  in  the  program. 

The  criteria  are  general  enough  to  allow  for  proper  tailoring  for  different  contractors, 

different  contract  types,  and  different  projects  (DoD  EVM  Guide,  1997:2).  The 

“generality”  provides  contractors  flexibility  to  develop  a  system  that  is  best  suited  to  their 

management  needs  (DoD  EVM  Guide,  1997:3).  The  criteria  are  utilized  to  assess 

whether  a  contractor  has  an  acceptable  EVM  system  (DoD  EVM  Guide,  1997:5). 
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What  Do  EVM  Systems  Really  Do? 

At  a  fundamental  level,  EVM  establishes  a  structure  for  management  to  complete 
the  steps  in  Table  10. 


Table  10.  Brief  Summary  of  EVM  Criteria  (DoD  EVM  Guide,  1997:5-8) 

1 .  Organization 

-  Define  the  work  to  be  performed 

-  Establish  the  program  structure  which  provides  for  integration 
of  planning,  scheduling,  budgeting,  and  control 

2.  Planning  and  Budgeting 

-  Schedule  the  work 

-  Identify  milestones,  goal  points  and  other  means  to  measure  progress 

-  Break  work  into  work  packages  and  establish  budgets  for  work  segments 

3.  Accounting  Considerations 

-  Record  costs  in  a  formal  system 

-  Track  control  account  budgets  to  organizational  elements 

4.  Analysis  and  Management  Reports 

-  Monthly  report  of  schedule  and  cost  variance 

-  Provide  detailed  explanations  for  significant  variances 

-  Complete  revised  estimates  resulting  from  performance  to  date 

5.  Revisions  and  Data  Maintenance 

-  Incorporate  authorized  changes  timely  (budget  and  schedule) 

-  Prevent  unauthorized  changes 

-  Document  changes  to  the  performance  measurement  baseline _ 


Important  to  this  study  is  the  understanding  of  how  EVM  tracks  work,  in  terms  of 
schedule  and  budget,  and  how  schedule  and  cost  variances  are  computed.  Figure  3 
defines  different  acronyms  used  in  the  EVMS  to  assess  schedule  and  cost  performance. 

Figure  4  provides  a  visual  depiction  how  cost  and  schedule  variances  are 
computed  for  the  analysis  and  management  reports  part  of  EVM.  This  study  focuses  on 
the  cost  variances  for  DoD  weapons  system  acquisition  programs. 
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BCWS 

Budgeted  Cost  for  Work  Scheduled 

PLANNED  VALUE 

BCWP 

Budgeted  Cost  for  Work  Performed 

EARNEDVALUE 

ACWP 

Actual  Cost  of  Work  Performed 

ACTUAL  COST 

BAC 

Budget  at  Completion 

EAC 

Estimate  at  Completion 

Figure  3.  EVM  Terminology  (Haupt,  2002) 


VARIANCES  (Favorable  is  positive,  Unfavorable  is  negative) 
•Cost  Variance  CV  =  BCWP  -  ACWP 

•Schedule  Variance  S V  =  BCWP  -  BCWS 

•  Variance  at  Completion  VAC  =  BAC-  EAC 


Figure  4.  EVM  Variance  Analysis  (DSMC  Gold  Card,  2000). 

The  complete  DSMC  Gold  Card  can  be  found  in  Appendix  A.  The  Gold  Card  provides 
detail  on  what  makes  up  the  Performance  Measurement  Baseline  (PMB)  and  a  pictorial 
view  of  how  work  is  broken  into  work  and  planning  packages.  It  is  important  to  note  that 
BCWP  =  BCWS  at  the  end  of  a  contract.  For  this  reason,  the  focus  of  this  analysis  is 
solely  on  cost  variance. 


CV  %  =  CV  / BCWP 
S V  %  =  sv/  BCWS 
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From  Figure  4,  it  becomes  apparent  why  EVM  is  such  a  valuable  tool  to  the 
program  manager.  At  any  point  in  time,  a  program  can  be  assessed  as  to  how  complete 
the  program  is  and  how  the  program  is  spending  budgetary  resources  compared  to  the 
plan. 

Summary 

This  section  provided  detail  as  to  how  EVM  came  to  be  and  provided  some  detail 
on  how  a  program  manager  uses  EVM  tools.  This  study  will  utilize  EVM  data  to  analyze 
how  cost  variances  have  changed  during  and  after  reform  and  to  determine  if  there  are 
discernable  time  lags  between  reform  and  cost  variance  perturbations. 

Past  Research 

This  section  summarizes  different  research  studies  related  to  effectiveness  of 
acquisition  reform  and  to  program  cost  growth  and  cost  variance.  These  studies  and  their 
results  motivated  this  study  and  provided  insight  into  methodology  and  analysis. 

Drezner  Study 

Drezner  et  al.  conducted  an  empirical  study  in  1993  to  quantify  the  magnitude  of 
cost  growth  in  weapon  systems  and  to  identify  factors  affecting  cost  growth  (Drezner  et 
al.,  1993:xi).  The  Drezner  study  utilized  Selected  Acquisition  Reports  (SAR)  data  to 
calculate  the  cost  growth  for  DoD  programs.  The  research  looked  at  SAR  data  for  197 
DoD  programs  as  of  December  1990,  and  found  that  little  improvement  has  occurred 
over  time  and  that  cost  growth  has  fluctuated  around  20  percent  since  the  mid  1960’s 
(Drezner  et  al.,  1993:xiii).  The  results  were  not  able  to  pinpoint  a  single  factor  that 
affected  cost  growth,  but  reasoned  “the  apparent  consistency  in  cost  growth  could  be 
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explained  in  terms  of  incomplete  or  incorrect  implementation  of  the  various  cost  control 
and  budgeting  initiatives,  due  to  strong  institutional  barriers”  (Drezner  et  al.,  1993:xiv). 
Results  also  highlighted  that  “until  later  in  this  decade  [1990-2000]  we  will  not  be  able  to 
detect  whether  initiatives  implemented  as  a  result  of  the  Packard  Commission  or  the  more 
recent  DMR  have  had  the  desired  effect”  (Drezner  et  al.,  1993:50). 

Searle  Study 

In  1997,  empirical  research  by  Searle  studied  the  impact  of  the  Packard 
Commission  on  reducing  cost  overruns  within  major  defense  acquisition  programs. 

Searle  analyzed  contracts  that  were  completed  between  1  January  1988  and  31  December 
1995  (Searle,  1997:45).  Searle  concluded  that  not  only  did  contract  cost  overruns  not 
improve  following  the  Packard  Commission,  but  also  they  worsened  in  the  years 
following  implementation  (Searle,  1997:76).  Specifically,  Searle  found  that  the 
worsening  effect  was  found  on  development  contracts  while  production  contracts 
experienced  no  significant  change  as  a  result  of  the  Packard  Commission 
recommendations  (Searle,  1997:82).  Searle’s  methodology  and  recommendations 
motivated  this  study. 

Coopers  and  Lybrand 

In  October  1997,  Coopers  and  Lybrand  conducted  an  opinion  survey  to  see  how 
effective  DoD  programs  were  at  implementing  acquisition  reform  initiatives  originated 
through  legislation  or  policy  changes,  at  the  contract  level  (Coopers  and  Lybrand, 

1997:2).  The  researchers  surveyed  10  defense  contractors  to  assess  implementation.  The 
survey  covered  90  contracts  valued  at  $17.4  billion  (Coopers  and  Lybrand,  1997:3).  In 
the  realm  of  costs,  the  survey  respondents  noted  significant  savings  in  elimination  of 
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military  specifications,  commercial  exemption  to  cost  or  pricing  data  and  using  an  open 
systems  approach  (Coopers  and  Lybrand,  1997:15).  The  Coopers  and  Lybrand  team 
concluded  that  “there  is  a  strong  empirical  basis  for  demonstrating  that  significant 
progress  has  been  made  over  the  past  four  years  in  the  implementation  of  acquisition 
reform  in  DoD  contracts”  (Coopers  and  Lybrand,  1997:7).  This  study  attempts  to 
determine  if  the  Coopers  and  Lybrand  assessment  of  “significant  progress”  is  true  from 
contract  cost  performance  point  of  view. 

Christensen  and  Templin  Study 

Christensen  and  Templin  (2002)  conducted  an  empirical  study  to  determine  if 
acquisition  reform  initiatives  spurred  by  the  Navy’s  A- 12  program  cancellation  (1991) 
had  impacted  cost  performance  (Christensen  and  Templin,  2002:105).  Christensen  and 
Templin  analyzed  240  defense  contracts  and  found  that  the  post  A- 12  time  period 
(defined  as  those  contracts  which  began  after  31  December  1991)  contracts  experienced 
improved  cost  performance  (Christensen  and  Templin,  2002:113-114).  The  study  also 
suggested  that  “transitional”  time  period  contracts  (contracts  started  before  reform  events 
and  finished  after  reform  events)  experience  worse  cost  performance  due  to  the 
turbulence  experienced  during  change  (Christensen  and  Templin,  2002:114). 

Chapter  Summary 

Chapter  2  provided  a  review  of  the  reform  initiatives  that  the  DoD  has  worked 
through  over  the  past  32  years.  This  review  is  a  fundamental  building  block  of  the  Phase 
Two  analysis  for  developing  the  timeline  of  acquisition  reform  events.  Acquisition 
Reform  is  nothing  new  to  the  DoD.  The  difference  today  is  a  reduced  budget,  changed 
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military  and  defense  environment,  and  new  leadership.  EVM  is  a  well-established 
management  tool  and  the  valuable  cost  variance  information  it  provides  on  DoD  contracts 
will  be  utilized  in  this  study.  The  prior  research  discussed  in  this  chapter  laid  the 
foundation  for  this  research  area,  motivated  the  research  objectives  and  clarified  the 
methodology  employed  in  this  study.  Chapter  3  describes  the  methodology  for  studying 
DoD  contracts  for  the  cost  variance  and  “rule  of  thumb”  analysis. 
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III.  Methodology 


Chapter  Overview 

This  chapter  explains  the  analysis  process  for  this  research  effort.  It  begins  with  a 
description  of  the  database  and  data  used,  and  explains  the  phases  of  analysis  conducted. 
Each  phase  is  broken  down  into  explanations  of  research  design,  population,  variables 
studied,  hypotheses  tested  and  how  the  data  was  prepared  for  each  analysis  phase. 

Data  Collection 

The  data  utilized  for  the  analysis  comes  from  the  Defense  Acquisition  Executive 
Summary  (DAES)  system.  The  DAES  was  established  in  1984  and  serves  as  an  internal 
Department  of  Defense  (DoD)  reporting  document  (AFMC  Financial  Management 
Handbook,  2001:38-12).  The  DAES  serves  two  primary  functions:  to  provide  progress 
information  to  DoD  leadership  on  Major  Defense  Acquisition  Programs  (MDAPs)  and  to 
provide  the  DoD  acquisition  community  feedback  (DAES  Web:l;  DoD  5000- 2R, 
200EC7.15.3). 

The  DAES  is  maintained  by  the  Office  of  the  Under  Secretary  of  Defense  for 
Acquisition,  Technology  and  Logistics  (OUSD[AT&L])  and  contains  cost  and  schedule 
performance  data  on  more  than  500  contracts  since  the  early  1970s  (Christensen  and 
Templin,  2000:197).  The  earned  value  management  data  used  in  this  study  is  taken  from 
the  DAES  database.  This  data  is  summarized  from  contractor  Cost  Performance  Reports 
(CPR)  by  government  program  offices  and  deemed  reliable  by  the  contractor’s 
requirements  under  the  EVM  system  criteria  described  in  Chapter  2  (Christensen  and 
Templin,  2000:197;  Christensen  and  Templin,  2002:108;  Gordon,  1996:33-34).  The 
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criteria  “are  internal  controls  intended  to  encourage  adequate  planning  and  control” 
(Christensen  and  Templin,  2002:108).  The  government  deems  the  performance  data 
reliable  when  the  contractor  is  compliant  with  EVM  system  criteria  (Christensen  and 
Templin,  2002:108). 

Data  from  the  DAES  used  for  this  study  is  current  as  of  June  2002.  The  DAES 
database  contains  multiple  fields  to  describe  each  contract  contained  within  the  system. 
For  the  purposes  of  this  study  the  following  fields  were  used  to  prepare  the  data  and 
perform  the  necessary  calculations  for  the  two  phases  of  analysis: 

•  Submit  Date  -  date  the  CPR  data  is  submitted  to  OUSD[AT&L]  for  inclusion 
into  the  database. 

•  Actual  Cost  of  Work  Performed  (ACWP)  -  cost  incurred  in  accomplishing  the 
work  performed. 

•  Budgeted  Cost  of  Work  Performed  (BCWP)  -  earned  value,  also  the  estimated 
cost  of  the  work  performed. 

•  Contract  Budget  Base  (CBB)  -  the  total  of  all  negotiated  cost  of  the  contract. 

•  Management  Reserve  (MR)  -  contractor’s  contingency  budget;  part  of  CBB. 

•  Contract  ID  (CID)  -  identifies  what  service  manages  the  contract. 

•  Contract  Type  (CTYPE)  -  identifies  whether  contract  is  cost  plus,  fixed  price 
or  some  combination/hybrid. 

•  Program  Phase  (PPHASE)  -  identifies  if  contract  is  for  production  or 
development  phase  of  program  life  cycle. 

Phase  One  Analysis 

Research  Design 

Chapter  2  detailed  the  history  of  acquisition  reform  and  the  timeline  of  the  current 
acquisition  reform  initiatives.  The  Federal  Acquisition  Streamlining  Act  (FAS A)  of  1994 
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marked  the  congressional  initiation  of  the  reform  effort  for  the  DoD,  however,  within  the 
department  the  reform  effort  was  already  under  way.  The  assignment  of  Colleen  Preston 
as  the  first  Deputy  Under  Secretary  of  Defense  for  Acquisition  Reform  (DUSD[AR])  in 
June  1993,  jump  started  DoD’s  reform  initiatives  (Munechika,  1997:31).  By  the  start  of 
fiscal  year  1995,  the  DoD  had  complied  with  all  applicable  components  of  Title  V  of  the 
FASA,  those  aimed  at  better  cost  control  goals  and  implementation  of  those  goals 
(GAO/NSIAD-97-22BR:2).  This  early  headway  and  initial  success  was  further 
evidenced  by  speeches  of  both  the  Kaminski  in  February  1997  and  Gansler  in  March 
1998  (Kaminski,  1997;  Gansler,  1998). 

This  study  will  utilize  a  cut-off  date  of  31  December  1997.  This  date  was  selected 
for  the  following  reasons: 

•  Passage  of  Clinger-Cohen  Act  of  1996  (late  1996),  which  mandated 
budget  for  training  and  education; 

•  DoD  5000  Series  update  released  in  1997;  and 

•  Speeches  by  Kaminski  and  Gansler  regarding  success  to  date  in  1997  and 
1998,  respectively. 

Consistent  with  the  Searle  study,  this  date  is  judgmentally  selected  (Searle,  1997:45). 

This  delineation  date  also  allows  for  four  years  of  analysis,  based  on  the  June  2002 
DAES  database,  on  each  side  of  the  cut-off  date  for  comparison  of  cost  performance. 

This  study  will  use  statistical  analysis  to  compare  the  before  (1  January  1994  - 
31  December  1997)  population  to  the  after  (1  January  1998  -  31  December  2001) 
population  to  assess  the  impact  of  the  current  reform  initiatives  on  cost  performance. 
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Research  Population 

The  previous  sections  laid  the  groundwork  for  defining  the  research  population 
for  Phase  One.  The  first  review  of  the  database  was  to  eliminate  any  contracts  that  did 
not  have  EVM  data,  both  BCWP  and  ACWP,  as  cost  performance  cannot  be  assessed 
against  contracts  without  this  information  (DSMC  Gold  Card,  2000).  The  pre-current 
reform  period  is  defined  as  1  January  1994  to  31  December  1997  and  the  post-current 
reform  period  is  defined  as  1  January  1998  to  31  December  2001.  Only  contracts  that 
completed  work  during  this  eight  year  time  period  are  included  in  this  phase.  Consistent 
with  the  Searle  study  in  1997,  the  database  was  further  divided  into  either  development  or 
production  phases  (Searle,  1997:46).  For  this  portion  of  Phase  One  analysis,  any  contract 
not  indicating  the  phase  was  eliminated  from  a  program  phase  comparison.  The  division 
scheme  for  program  phase  is  defined  in  Table  11.  Any  contracts  that  were  not  identified 
by  any  of  the  terms  in  Table  11  were  eliminated  from  the  program  phase  comparison,  as 
were  contracts  that  contained  multiple  terms  from  both  the  production  phase  and 
development  phase  columns. 


Table  11.  Program  Phase  Division  Identifiers 


Production  Phase 

Development  Phase 

Production 

Ship  Construct 

LRIP 

Procurement 

Prod  and  Deploy 

Fabrication 

PROD 

Development 

Full  Scale  Development  (FSD)  or  (FSED) 

Engineering  &  Manufacturing  Development  (EMD) 
Engineering  Development 

Research  Development  (RDT&E) 

DEV 

Sys  Dev 

Similar  to  the  program  phase  comparison,  the  data  was  divided  by  contract  type, 
either  cost  plus  or  fixed  price.  This  portion  of  the  analysis  is  similar  to  the  program 
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phase,  but  considers  reform  impact  based  on  contract  type.  This  research  expects  to  see  a 
significant  change  for  both  contract  types  as  all  contracts  were  equally  affected  by  the 
FASA  (see  Chapter  2,  Table  6)  (DSMC  FASA).  Therefore,  any  contracts  that  did  not 
contain  information  on  contract  type,  or  contained  a  hybrid  (e.g.  FPEAF/CPAF),  were 
excluded  from  this  portion  of  the  analysis. 

Completed  Contract  Defined 

The  DAES  database  does  not  contain  earned  value  data  for  all  contracts  at  the 
100%  completion  point  (Searle  1997:47;  Christensen  and  Templin,  2002:108).  Prior 
research  indicates  that  at  the  75%  completion  point,  contract  cost  performance  remains 
relatively  stable  (Gordon,  1996:Ch  3;  Searle,  1997:47;).  Searle  points  out  that  “the 
contract  is  nearly  complete  at  this  point  [75%  and  greater]  in  terms  of  the  costs  incurred” 
(Searle,  1997:47).  This  study  defines  completion  in  the  same  manner.  The  completed 
contract  cost  data  used  in  the  analysis  is  the  last  report  submittal  date  for  contracts  at  75% 
or  greater  and  meet  the  cut-off  dates  defined  previously  for  pre-  and  post-reform  time 
periods. 

The  DAES  database  does  not  contain  a  field  that  defines  percent  completion. 

This  value  must  be  calculated  using  the  data  available.  Equations  1  and  2  are  used  to 
calculate  percent  complete.  The  Contract  Budget  Base  (CBB)  contains  the  Budget  at 
Complete  (BAC),  which  is  the  total  cost  of  the  planned  work,  and  Management  Reserve 
(MR).  The  percent  complete  simply  relates  the  current  amount  of  work  complete,  or 
BCWP,  to  the  total  amount  of  work  on  the  contract,  represented  by  the  BAC  (DSMC 
Gold  Card,  2000). 
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BAC  =  CBB  -  MR  (1) 


Percent  Complete  =  BCWP  /  BAC  (2) 


At  this  point,  the  database  is  ready  for  the  necessary  analysis  to  reduce  it  to  the  two 
samples  necessary  for  the  Phase  One  analysis. 

Sample  Defined 

The  samples  utilized  for  this  study  included  all  DAES  database  contract 
information  that  meets  all  of  the  following  requirements: 

•  Completion  (75%  or  greater)  as  of  the  cut-off  dates  for  pre-  and  post¬ 
acquisition  reform  periods 

•  Defined  as  either  production  or  development  (as  defined  in  Table  1 1)  for 
production  and  development  program  phase  analysis 

•  Defined  as  either  a  Cost  Plus  or  Fixed  Price  contract  type  for  contract  type 
analysis 

Table  12  provides  statistics  on  the  samples  available  from  the  DAES  database  meeting  all 
the  listed  requirements  and  used  for  this  study.  The  numbers  represent  number  of 
contracts  utilized  in  each  period  and  portion  of  the  analysis. 


Table  12.  Study  Sample  Statistics 


Group 

Pre-Reform 

Post-Reform 

(1  Jan  94  -  31  Dec  97) 

(1  Jan  98-31  Dec  01) 

All  DoD  contracts 

109 

95 

Proaram  Phase 

Production 

Development 

61 

36 

34 

44 

Contract  Tvoe 

40 

59 

60 

29 

Cost  Plus 
Fixed  Price 
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The  analysis  uses  all  contracts  for  each  part  of  the  Phase  One  analysis.  For  example, 
comparing  all  contracts,  109  pre-reform  completed  contracts  are  compared  against  95 
post-reform  completed  contracts.  For  the  program  phase  portion  of  the  analysis,  10  of  the 
109  pre-reform  contracts  and  17  of  the  95  post- reform  contracts  are  removed  because  all 
27  of  the  excluded  contracts  do  not  contain  distinct  (Table  11)  program  phase 
information.  For  the  contract  type  portion  of  the  analysis,  10  of  the  109  pre-reform 
contracts  and  6  of  the  95  post-reform  contracts  are  excluded  because  all  16  of  the 
excluded  contracts  do  not  contain  specific  contract  type  information.  In  these  cases, 
either  there  is  no  contract  type  data  or  the  contract  type  is  a  fixed  price/cost  plus  hybrid. 

Research  Variable 

This  study  is  concerned  with  analyzing  the  impact  of  acquisition  reform  on  cost 
performance.  The  final  overrun  (FO)  of  each  sample  contract  is  calculated  with 
Equation  3.  This  equation  takes  the  actual  cost  of  work  performed  (ACWP)  data  and 
compares  it  to  the  budgeted  cost  of  work  performed  (BCWP).  Both  the  ACWP  and  the 
BCWP  values  used  in  this  analysis  are  the  final  reported  values  for  all  sample  contracts. 

FO  =  ACWP  -  BCWP  (3) 

As  a  result  of  the  equation  used  for  this  analysis,  all  overruns  will  compute  as  a  positive 
number,  which  is  contrary  to  how  overruns  are  reported  in  the  EVM  system  and 
associated  reports.  This  is  important  to  note  for  understanding  the  results  of  the 
hypotheses  testing  and  for  Phase  Two  analysis.  Therefore,  a  negative  FO  value  would 
indicate  a  cost  underrun  for  the  completed  contract  in  the  sample. 
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The  comparison  of  final  overruns  or  underruns  does,  however,  have  one  problem. 
A  larger  dollar  value  contract  would  overshadow  performance  on  a  smaller  dollar  value 
contract.  Therefore,  the  data  must  be  normalized  in  order  to  do  conduct  a  reliable 
comparison.  This  adjustment  calculates  the  final  overrun  percentage  (FO%),  as  this 
statistic  will  have  the  same  meaning  regardless  of  the  size  of  the  contract  studied. 
Equation  4  illustrates  how  this  is  computed.  As  noted  previously,  a  negative  percentage 
indicates  a  final  underrun  percentage.  The  use  of  the  final  overrun  percentage  equation 
takes  the  final  overrun,  as  calculated  in  Equation  3,  and  divides  this  value  by  the  BCWP. 

FO%  =  Final  Overrun  (FO)  /  BCWP  (4) 

To  perform  the  comparison  of  the  cost  performance  for  the  pre-  and  post-reform 
population  samples,  this  study  uses  the  mean  FO%  of  each  sample  as  the  study 
parameter.  This  is  computed  using  the  equation  given  in  Equation  5  (Searle,  1997:51). 


FO% 


(  n 


Y^FO% 


/  n 


(5) 


i  =  the  ith  contract  in  population 
n  =  total  number  of  contracts  in  population 


Research  Question  and  Hypotheses 

Research  question:  Did  the  current  acquisition  reform  initiatives  have  an  effect 
on  contract  cost  performance?  To  answer  this  question,  the  following  five  hypotheses  are 
tested: 
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Hypothesis  1 :  Is  the  mean  F0%  for  all  DoD  pre-reform  (1  January  1994  - 
31  December  1997)  contracts  different  than  the  mean  FO%  for  all  DoD  post- reform 
(1  January  1998  -  31  December  2001)  contracts? 

Ho-  mean  FO%pre-reform  =  mean  FO%post -reform 
Ha-  mean  FO%pre-reform  ^  mean  FO%post -reform 

If  the  null  hypothesis  is  true,  this  would  indicate  that  the  current  reform  period  did  not 

have  a  significant  impact  on  cost  performance.  If  the  alternate  hypothesis  is  true,  this 

would  indicate  the  opposite,  that  the  current  reform  period  did  have  a  significant  impact 

on  cost  performance. 

Hypotheses  2  and  3:  Is  the  mean  FO%  for  DoD  pre-reform  (1  January  1994  - 
31  December  1997)  contracts  different  than  the  mean  FO%  for  DoD  post-reform 
(1  January  1998  -  31  December  2001)  contracts  in  the  development  or  production  phase? 

Ho-  mean  FO%pre-reform  (development  phase)  =  mean  FO%post-reform  (development  phase) 

Ha.  mean  FO%pre-reform  (development  phase)  ^  mean  FO%post -reform  (development  phase) 

Ho.  mean  FO%pre-reform (production  phase)  =  mean  FO%post-reform  (production  phase) 

Ha.  mean  FO%pre-reform  (production  phase)  ^  mean  FO%post -reform  (production  phase) 

If  the  null  hypothesis  is  true,  this  would  indicate  that  the  current  reform  period  did  not 
have  a  significant  impact  on  cost  performance  for  development  or  production  contracts, 
respectively.  If  the  alternate  hypothesis  is  true,  this  would  indicate  the  opposite,  that  the 
current  reform  period  did  have  a  significant  impact  on  cost  performance  for  development 
or  production  contracts,  respectively. 

.Hypotheses  4  and  5:  Is  the  mean  FO%  for  DoD  pre-reform  (1  January  1994  - 
31  December  1997)  contracts  different  than  the  mean  FO%  for  DoD  post-reform 
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(1  January  1998  -  31  December  2001)  contracts  based  on  contract  type  (cost  plus  and 
fixed  price)? 

Ho-  mean  FO%pre-reform  (cost  plus  contracts)  =  mean  FO%post-reform  (cost  plus  contracts) 

Ha.  mean  FO%pre-reform  (cost  plus  contracts)  ^  mean  FO%post -reform  (cost  plus  contracts) 

Ho-  mean  FO%pre-reform  (fixed  price  contracts)  =  mean  FO%post-reform  (fixed  price  contracts) 

Ha-  mean  FO%pre-reform  (fixed  price  contracts)  ^  mean  FO%post  -reform  (fixed  price  contracts) 

If  the  null  hypothesis  is  true,  this  would  indicate  that  the  current  reform  period  did 
not  have  a  significant  impact  on  cost  performance  for  cost  plus  or  fixed  price  contracts, 
respectively.  If  the  alternate  hypothesis  is  true,  this  would  indicate  the  opposite,  that  the 
current  reform  period  did  have  a  significant  impact  on  cost  performance  for  cost  plus  or 
fixed  price  contracts,  respectively. 

Statistical  Analysis 

The  Phase  One  analysis  hypotheses  all  involve  comparing  mean  values  and 
testing  to  see  if  they  are  statistically  similar  or  different.  The  first  step  in  conducting  the 
tests  is  to  decide  whether  to  use  a  parametric  or  nonparametric  test.  Parametric  tests 
make  specific  assumptions  with  regard  to  the  population  parameters  that  characterize  the 
underlying  distributions  for  which  the  test  is  employed  (Sheskin,  2000:33). 

Nonparametric  tests  make  no  assumptions  about  the  population  parameters  (Sheskin, 
2000:33). 

The  t-test  for  two  independent  samples  involves  using  the  two  sample  means  to 
estimate  the  values  of  the  means  of  the  populations  from  which  they  come  (Sheskin, 
2000:247).  If  the  result  of  the  t-test  is  significant,  the  conclusion  will  be  that  “there  is  a 
high  likelihood  that  the  samples  represent  populations  with  different  means”  (Sheskin, 
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2000:247).  The  following  assumptions  regarding  the  sample  data  must  be  met  to  use  the 
two- sample  t-test:  the  sample  data  has  been  randomly  selected  from  the  population  and 
the  populations  are  independent  of  each  other;  the  population  data  comes  from  a  normal 
distribution;  and  the  populations  must  have  the  same  variance  (Sheskin,  2000:247). 
Should  any  of  the  assumptions  required  for  use  of  the  t-test  be  violated,  it  is  possible  that 
any  results  generated  from  conducting  the  two-sample  t-test  may  be  inaccurate  or 
compromised  (Sheskin  2000:247). 

The  analogous  nonparametric  test  to  the  two- sample  t-test  is  the  Mann- Whitney  U 
test;  useful  when  the  normality  assumption  is  violated  (Devore,  2000:659;  Searle, 
1997:55;  Sheskin,  2000:269).  Research  has  shown  that  the  Mann- Whitney  U  test 
provides  a  powerful  test  option  when  the  normality  assumption  has  been  violated 
(Sheskin,  2000:297).  Many  statisticians  consider  the  Mann- Whitney  U  test  the  best 
nonparametric  test  for  location  (Gibbons,  1971:149;  Searle,  1997:56).  The  Mann- 
Whitney  U  test  is  95%  as  efficient  in  discerning  significant  population  differences 
compared  to  the  two-sample  t-test  when  the  normality  assumption  is  met,  but  is  over 
100%  as  efficient  in  doing  the  same  when  the  normality  assumption  is  not  met  (Sheskin, 
2000:297). 

Empirical  evidence  shows  that  the  two-sample  t-test  is  reasonably  robust, 
meaning  it  provides  reliable  information  about  the  underlying  sampling  distribution, 
despite  violation  of  any  of  the  assumptions  (Sheskin,  2000:269).  Researchers  argue  that 
parametric  tests,  the  two-sample  t-test  in  particular,  are  more  powerful  than  their 
analogous  nonparametric  tests  (Sheskin,  2000:270).  Previous  research  shows  that  cost 
variances  are  not  normally  distributed;  therefore,  both  the  two-sample  t-test  and  the 
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Mann- Whitney  U  test  will  be  used  to  ensure  completeness  (Christensen  and  Templin, 
2002:108). 

For  all  hypothesis  testing,  a  significance  level  (alpha  value  [a])  of  .05  will  be 
used.  The  results  will  be  presented  with  the  appropriate  test  statistic  value,  but  will  be 
explained  using  p- values.  The  p- value  is  the  smallest  level  of  significance  at  which  the 
null  hypothesis  ( H„ )  would  be  rejected  when  a  specified  test  procedure  is  used  (Devore, 
2000:342).  The  p- value  is  then  compared  against  the  alpha  level  (.05)  to  either  accept  or 
reject  the  null  hypothesis  (Devore,  2000:342).  Table  13  displays  the  acceptance  and 
rejection  criteria  for  the  tests  using  p- values  (Devore,  2000:342).  The  smaller  the  p- 
value,  the  more  contradictory  is  the  data  to  the  null  hypothesis  (Devore,  2000:342).  The 
benefit  of  p-values  is  that  they  convey  information  about  the  strength  of  evidence  against 
the  null  hypothesis  (Ha)  and  allow  a  decision  maker  to  draw  conclusions  at  any  specified 
level  of  significance  (a)  (Devore,  2000:340). 

Table  13.  Hypothesis  Test  Decision  Table 

P- value  <  .05  =>  reject  Ho  at  level  .05 

P- value  >  .05  =>  do  not  reject  H0  at  level  .05 

Data  Preparation 

Prior  to  conducting  hypothesis  testing,  the  assumptions  of  the  two -sample  t-test 
and  the  Mann- Whitney  U  test  will  be  determined.  The  sample  data  must  be  analyzed  to 
assess  independence,  randomness,  normality  and  constant  variance  (two-sample  t-test 
only).  The  purpose  of  this  assessment  is  to  determine  beforehand  which  testing  method 
is  most  appropriate  to  aid  in  interpreting  the  results  in  Chapter  4. 
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Independence 

In  the  Searle  study  (1997),  independence  was  assumed  based  on  the  following: 

•  Likeness  caused  by  legislation  and  regulation  would  affect  cost  performance 
equally  across  all  DoD  contracts;  and 

•  Multiple  contracts  under  similar  program  managers  are  run  by  a  multitude  of 
cost  management  contractors  and  personnel  (Searle,  1997:58-59). 

The  characteristics  of  the  populations  utilized  in  this  study,  and  their  respective  samples, 

are  the  same.  All  DoD  contracts  are  managed  under  the  same  legislation  and  regulations. 

In  fact,  the  point  of  this  study  to  see  if  changes  to  the  legislation  and  regulations  impacted 

cost  performance  by  comparing  pre-  and  post- reform  populations.  Within  each 

population,  there  are  a  multitude  of  support  contractors  and  financial  management 

personnel,  with  varying  degrees  of  experience  and  training,  performing  cost  management 

activities.  For  these  reasons,  independence  of  the  populations  can  be  assumed.  However, 

it  is  important  to  acknowledge  the  possibility  that  the  samples  do  not  satisfy  the 

independence  requirement,  which  could  invalidate  the  results  of  the  analysis. 

Random  Sample 

Random  sample  is  defined  as  “one  for  which  any  particular  subset  of  the  specified 
size  has  the  same  chance  of  being  selected”  (Devore,  2000:8).  In  this  study,  all 
applicable  population  data  contained  in  the  DAES  database  is  used  in  the  analysis. 
However,  because  “applicable  data”  contains  contracts  that  have  all  the  necessary  EVM 
data  reported,  the  sample  cannot  be  true  random  samples.  The  population  samples 
contain  all  available  data  points  contained  in  the  DAES  database,  therefore,  the  random 
sample  requirement  is  assumed.  However,  it  is  critical  to  note  that  without  the  random 
sample  assumption  satisfied,  the  results  of  this  study  would  be  questionable. 
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Normality  and  Equality  of  Variance 

Normality  will  be  tested  both  subjectively  and  objectively.  The  subjective 
assessment  will  consist  of  normal  probability  plots.  According  to  D’Agostino  and 
Stephens,  normal  probability  plotting  should  always  accompany  a  formal  test  of 
normality  (D’Agostino  and  Stephens,  1986:405).  A  straight  line  would  indicate  the 
existence  of  an  underlying  normal  distribution  for  the  sample  data  (D’Agostino  and 
Stephens,  1986:35).  The  graphical  representation  serves  only  as  an  informal  preliminary 
judgment  of  normality  and  supports  formal  numerical  techniques  (D’Agostino  and 
Stephens,  1986:41). 

The  Shapiro-Wilk  test  for  normality  will  serve  as  the  objective  assessment.  The 
Shapiro-Wilk  test  is  a  powerful  non- graphical  test  for  the  assumption  of  normality 
(D’Agostino  and  Stephens,  1986:406;  Searle,  1997:60;  Verrill,  1981:3).  The  Shapiro- 
Wilk  test  p-  value  will  also  indicate  the  significance  of  any  violations  of  the  normality 
assumption.  The  null  hypothesis  of  this  test  is  that  the  data  comes  from  a  normal 
distribution  (Searle,  1997:66;  White,  2002).  Therefore,  any  p- values  less  than  or  equal  to 
.05  would  indicate  a  non- normal  distribution. 

The  test  for  equal  variances  involves  a  hypothesis  test  using  the  F  distribution 
(Devore,  2000:393).  The  null  hypothesis  is  that  the  population  variances  are  equal.  The 
test  statistic  is  computed  using  the  ratio  of  the  sample  variances  (Devore,  2000:393; 
Searle,  1997:60).  Again,  the  p- value  will  be  used  to  assess  either  the  acceptance  or 
rejection  of  the  equal  variance  null  hypothesis,  with  a  significance  level  of  .05.  The 
Levene  Test  for  Equality  of  Variances  is  used  in  this  study  because  this  test  is  less 
sensitive  to  departures  from  normality  (Neter,  Kutner,  Nachtsheim,  and  Wasserman, 
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1996:112;  NIST/SEMATECH,  2003).  This  is  important  as  prior  research  indicates  that 
cost  variance  data  tends  to  be  non-normally  distributed  (Christensen  and  Templin, 
2002:108).  The  results  of  the  Levene  Test  will  dictate  the  specific  t  test  employed  for  the 
parametric  hypothesis  testing.  If  the  test  indicates  equal  variances,  the  pooled  t  test  will 
be  selected  as  this  outperforms  the  two -sample  t  test  by  reducing  the  probability  of 
rejecting  the  null  hypothesis  when  null  hypothesis  is  true  (Devore,  2000:370). 

Conversely,  when  the  Levene  test  results  indicate  unequal  variances,  the  standard  two- 
sample  t  test  procedures  will  be  selected  as  the  pooled  t  test  can  lead  to  erroneous 
conclusions  if  applied  when  the  variances  are  different  (Devore,  2000:370). 

Phase  One  Summary 

This  section  detailed  the  Phase  One  analysis  methodology.  The  treatment  date 
and  study  populations  and  samples  were  defined,  research  variable  and  hypotheses 
explained,  and  methods  of  testing  detailed.  The  tests  of  assumptions  are  conducted  to 
indicate  which  test  method  is  most  appropriate.  For  all  hypotheses,  both  the  parametric 
two- sample  t-test  and  nonparametric  Mann- Whitney  U  test  will  be  utilized  for 
completeness. 

Phase  Two  Analysis 

Research  Design 

The  second  phase  of  analysis  is  focused  on  discovering  the  existence  of  any 
trends  or  time  lags  between  acquisition  reform  initiatives  and  cost  performance  changes, 
either  positive  or  negative.  This  phase  does  not  include  statistical  tests,  but  simply  plots 
the  cost  performance  of  all  active  contracts  over  the  time  period  provided  in  the  DAES 
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database  and  determines  if  consistent  trends  or  time  lags  exist  in  an  effort  to  develop  a 
“rule  of  thumb”.  The  research  conducted  for  Chapter  2  of  this  report  provides  sufficient 
information  to  develop  a  timeline  of  acquisition  reform  initiatives,  starting  with  Packard 
in  1970  and  concluding  with  the  Clinger-Cohen  Act  in  1996.  The  DAES  database 
contains  earned  value  data  for  contracts  from  1970  to  June  2002  for  the  cost  performance 
assessment. 

Research  Population  and  Sample 

The  research  population  consists  of  all  contracts  reported  in  the  DAES  database. 
The  sample  selected  from  the  population  includes  all  contracts  that  provide  earned  value 
data.  Any  contract  with  incomplete  earned  value  data  was  eliminated  from  this  analysis 
because  cost  performance  cannot  be  assessed.  The  analysis  used  all  available  contracts 
because  all  contracts  are  managed  under  the  same  regulations  and  legislation  and  the 
DoD  and  Congress  are  concerned  with  all  DoD  contracts,  not  solely  with  any  particular 
service. 

Similar  to  Phase  One,  the  analysis  will  also  look  at  program  phase  and  contract 
type  to  see  if  reform  initiatives  affect  these  contracts  differently.  The  division  scheme 
utilized  is  exactly  the  same  as  performed  in  phase  one.  Any  contracts  from  the  “all 
contracts”  sample  explained  in  the  prior  paragraph  that  did  not  contain  information  on 
specific  program  phase  (Table  1 1)  or  specific  contract  type  (either  fixed  price  or  cost 
plus)  were  eliminated  from  this  portion  of  the  phase  two  analysis. 

Research  Variable 

Phase  Two  analysis  focused  on  cost  performance  over  time  by  looking  at  DoD’s 
overall  contract  cost  performance  at  snapshots  over  time.  The  snapshots  are  determined 
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by  the  DAES  submittal  dates  as  reported  in  the  DAES  database.  For  each  submittal  date, 
the  mean  ACWP  and  BCWP  are  calculated.  Equation  6  illustrates  how  these  are 
calculated.  The  mean  ACWP  and  mean  BCWP  are  used  to  account  for  the  inconsistent 
number  of  contracts  reported  in  different  submittal  periods,  as  well  as  adjust  for  periods 
when  larger  contracts  and/or  smaller  contracts  are  reported  (same  for  Phase  One).  These 
calculations  also  give  values  for  the  average  amount  of  work  completed  to  date  compared 
against  the  average  cost  of  the  same  work.  Utilizing  the  mean  helps  eliminate  spikes 
caused  by  a  period  of  larger  contracts  or  a  period  of  smaller  contracts  and  provides  a 
measure  of  the  center  for  the  ACWP  and  BCWP  (Makridakis,  Wheelwright,  and 
Hyndman,  1998:29). 


f 


ACWP,  = 


j>cw> 


V 


/  n 


(6) 


i  =  the  ith  submittal  in  month  j 
n  =  total  number  of  submittals  in  month  j 


The  variable  used  in  this  analysis  phase  is  the  mean  cost  overrun  percentage  for 
each  year.  Equations  7,  8,  and  9  illustrate  the  necessary  calculations.  This  mean 
accounts  for  differences  in  reporting  from  year  to  year  and  provides  a  measure  of  the 
center  of  the  data  (Makridakis,  Wheelwright,  and  Hyndman,  1998:29).  In  the  DAES 
database,  some  years  have  eleven  months  of  reporting,  while  other  years  have  four. 
Similar  to  the  interim  calculation  in  Equation  6,  this  averaging  smoothes  out  any  monthly 
spikes  caused  by  the  inconsistent  reporting  contained  in  the  database.  Again,  it  also 
provides  values  for  the  average  amount  of  work  completed  (mean  BCWP)  compared 
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against  the  average  cost  of  that  work  (mean  ACWP).  The  mean  cost  overrun  percentage 
is  calculated  as  a  positive  number  for  overruns  and  a  negative  number  for  underruns;  the 
same  procedure  used  in  Phase  One.  This  method  was  selected  for  the  appearance  of  the 
time  plots. 


ACWP,. 


2^  ACWP j 


/  m 


(7) 


j  =  the  jth  monthly  submittal  in  year  k 
m  =  total  number  of  monthly  submittals  in  year  k 


CO,  =  ACWPk  -  BCWPk  (8) 


CO%k  =COkl  BCWPk  (9) 


k  =  the  kth  year  of  DAES  reporting 


Trend  Versus  Timeline  Analysis 

The  most  obvious  graphical  form  for  time  series  data  is  the  time  plot  in  which  the 
data  are  plotted  over  time  (Makridakis,  Wheelwright,  and  Hyndman,  1998:24).  A  time 
plot  will  reveal  any  trends  over  time  (Chatfield,  1996:11;  Makridakis,  Wheelwright,  and 
Hyndman,  1998:24).  The  goal  is  to  see  if  a  “rule  of  thumb”  can  be  determined  based  on 
visually  examining  cost  performance  over  time  against  the  backdrop  of  acquisition 
reform  initiative  implementation. 

A  forecasting  concept  that  is  useful  in  describing  any  noted  trends  is  intervention 
analysis.  Interventions  are  one-time  events  that  have  an  impact  on  a  variable  (mean  cost 
overrun  %)  (Makridakis,  Wheelwright,  and  Hyndman,  1998:418).  The  intervention  is 
assumed  to  occur  at  one  time  period  only,  but  the  impact  can  be  instantaneous  or  spread 
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over  time  (Makridakis,  Wheelwright,  and  Hyndman,  1998:418).  Acquisition  reform 
initiatives  represent  interventions;  either  of  DoD  leadership.  Congress,  or  both,  to  impact 
the  way  the  defense  community  acquires  weapon  systems.  There  are  four  types  of 
interventions  that  will  be  considered  in  this  analysis.  They  are  step-based  interventions, 
pulse-based  interventions,  decayed  response  interventions,  and  delayed  response 
interventions  (Makridakis,  Wheelwright,  and  Hyndman,  1998:419-422).  Figure  5 
graphically  shows  what  each  of  these  interventions  might  look  like  (Makridakis, 
Wheelwright,  and  Hyndman,  1998:420). 


Figure  5.  Intervention  Graphical  Representation 


Step-based  interventions  are  where  there  is  a  sudden  and  lasting  drop  or  rise  in  the 
variable  (Makridakis,  Wheelwright,  and  Hyndman,  1998:419).  A  pulse-based 
intervention  affects  the  variable  at  the  point  of  intervention  but  nowhere  else 
(Makridakis,  Wheelwright,  and  Hyndman,  1998:422).  A  decayed  response  intervention 
has  an  immediate  impact  but  then  decays  away  slowly  (Makridakis,  Wheelwright,  and 
Hyndman,  1998:422).  Delayed-response  interventions  will  cause  a  rise  or  fall  in  the 
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variable,  but  not  instantaneously  (Makridakis,  Wheelwright,  and  Hyndman,  1998:421). 
The  goal  in  this  analysis  will  be  to  determine  if  there  is  a  common  intervention 
description  for  acquisition  reform  initiatives  and  the  time  period  associated  with  the 
impact. 

Phase  Two  Summary 

Phase  Two  analysis  is  focused  on  analyzing  the  cost  performance  of  DoD 
contracts,  in  total,  by  phase  and  contract  type.  This  will  be  accomplished  with  the  use  to 
time  plots  of  the  mean  cost  overrun  percentage  versus  time  (1970  -  2002).  The  goal  is  to 
identify  any  existence  of  trends  between  acquisition  reform  initiatives  and  cost 
performance  change.  The  goal  is  also  to  place  a  time  period  with  any  consistent  pattern 
of  intervention  impact. 

Phase  Three  Analysis 

Phase  Three  analysis  will  only  be  conducted  if  results  from  Phase  One  and  Phase 
Two  indicate  a  new  treatment  date  for  hypothesis  testing  should  be  considered.  The 
hypotheses  tested  and  hypothesis  testing  procedures  would  be  identical  to  Phase  One. 

The  only  differences  would  be  in  population  and  sample  for  pre-  and  post-reform  periods. 
Statistics  regarding  the  samples  and  treatment  date  selected  will  be  represented  in 
Chapter  4,  if  Phase  Three  becomes  necessary. 

Chapter  Summary 

This  chapter  explained  the  methodology  employed  to  conduct  the  phases  of 
analysis.  Phases  One  and  Three  focus  on  conducting  hypothesis  tests  on  pre-reform  and 
post-reform  contracts  to  see  if  there  is  a  significant  difference  in  cost  performance.  Phase 
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Two  focuses  on  visually  analyzing  DoD  contract  cost  performance  over  time  to  identify 
any  trends  and  time  lags  between  the  implementation  of  acquisition  reform  initiatives  and 
cost  performance  changes.  Chapter  4  provides  the  analysis  results. 
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IV.  Results 


Chapter  Overview 

This  chapter  presents  the  results  of  the  phase  analyses.  The  results  of  all 
statistical  tests  and  graphical  analysis  are  presented  by  each  individual  phase. 

Phase  One 

The  purpose  of  Phase  One  was  to  test  to  see  if  a  difference  existed  between  pre¬ 
reform  contracts  and  post-reform  contracts  by  performing  hypothesis  tests  of  the  mean 
final  cost  overrun  percentage  of  each  sample.  The  treatment  date  used  for  Phase  One  was 
31  December  1997.  The  analysis  consisted  of  five  hypothesis  tests;  one  test  for  all 
contracts,  one  for  fixed  price  contracts,  one  for  cost  plus  contracts,  one  for  production 
phase  contracts,  and  one  for  development  phase  contracts.  Table  12  provides  data  on  the 
number  of  contracts  used  in  each  hypothesis  test.  The  Phase  One  results  are  separated  by 
each  applicable  hypothesis  test. 

Prior  to  any  hypothesis  testing  of  the  means,  the  samples  were  tested  to  determine 
which  hypothesis  testing  method,  either  parametric  or  non- parametric,  was  more 
appropriate.  If  a  sample  appears  to  be  normally  distributed,  then  that  sample  is  tested  to 
assess  constant  variance.  Both  assumptions  are  essential  for  the  two-sample  t-test. 
Regardless  of  the  assumption  test  results,  both  the  parametric  (two-sample  t-test)  and 
non-parametric  (Mann- Whitney  U  Test)  are  used.  The  assumption  test  results  simply 
give  credence  to  which  mean  hypothesis  testing  method  is  more  credible.  All  normal 
probability  plots  are  found  in  Appendix  B  and  the  Shapiro- Wilk  test  values  are  found  in 
each  test  results  section. 
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A  statistical  software  package,  JMP  4.0,  calculated  all  statistical  test  values  used 
in  this  study.  The  software  uses  the  non-parametric  Wilcoxon- Mann- Whitney  test  for 
testing  the  means.  The  Mann- Whitney  test  is  equivalent  to  the  Wilcoxon- Mann- Whitney 
(a.k.a.  the  Wilcoxon  Rank-Sum  test)  used  by  the  JMP  4.0  software  (Devore,  2000:659; 
Gibbons,  1985:148;  Sheskin,  2000:313).  The  p-values  are  the  same  for  both  tests. 

The  results  for  the  two-sample  t  test  are  listed  for  each  test.  In  cases  where  the 
Levene  test  for  equal  variances  indicates  that  the  variances  are  not  equal,  the  Welch 
ANOVA  test  p- value  is  used.  The  Welch  ANOVA  is  equivalent  to  the  unequal  variance 
two- sample  t  test  when  comparing  two  sample  means  (JMP  4.0,  2001).  Conversely, 
when  the  Levene  Test  indicates  that  the  variances  are  equal,  the  equal  variance  two- 
sample  t  test  (pooled  t  procedure)  p- value  is  used. 

Hypothesis  Test  #1  -  All  DoD 

The  first  hypothesis  tested  to  determine  if  the  mean  final  cost  overrun  percentage 
(FO%)  of  all  pre-reform  contracts  (1  Jan  94-31  Dec  97)  is  significantly  different  than 
the  mean  FO%  of  all  post-reform  contracts  (1  Jan  98-31  Dec  01). 

Normality  and  Equal  Variance  Test  Results 

The  Shapiro-Wilk  test  conducted  on  the  pre-reform  sample  produced  a  test 
statistic  value  of  .81 19  and  a  p- value  of  .0000.  These  values  indicate  a  non- normal 
distribution  for  the  pre-reform  sample.  The  Shapiro-Wilk  test  conducted  on  the  post¬ 
reform  sample  produced  a  test  statistic  value  of  .6675  and  a  p- value  of  .0000.  These 
values  indicate  a  non-normal  distribution  for  the  post-reform  sample.  At  a  significance 
level  of  .05,  both  results  indicate  both  samples  are  non- normal;  therefore  more  emphasis 
will  be  placed  on  the  Mann- Whitney  test  results  over  the  two-sample  t-test. 
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The  Levene  test  for  equal  variances  produced  a  test  statistic  value  of  2.9127  and  a 
p-value  of  .0894.  This  result  indicates  that  the  population  variances  are  equal  at  a 
significance  level  of  .05.  Therefore,  the  pooled  t  test  results  will  be  utilized  in  the 
analysis. 

Hypothesis  Test  Results 

The  results  of  the  hypothesis  test  are  represented  in  Table  14.  The  test  statistic 
values  and  associated  p- values  are  consistent.  At  a  significance  level  of  .05,  the  results 
indicate  that  the  null  hypothesis  is  not  rejected. 


Table  14.  Phase  One  -  Hypothesis  Test  #1  Results 


All  DoD  contracts 

Mean  FO% 

Std  Error 

Pre-Reform 

8.2522 

1.399 

Post-Reform 

6.2478 

1.498 

Test  Results 

Test  Statistic  (Ws) 
P-value 

Two-SamDle  t 

-0.978 

0.3293 

The  Mann- Whitney  results  indicate  that  the  two  sample  distributions  do  not  differ 
in  location  and  the  two-sample  t-test  indicates  the  mean  FO%  for  each  sample  is  not 
significantly  different.  Both  results  suggest  that  the  mean  FO%  of  the  pre-reform  sample 
is  equal  to  the  mean  FO%  of  the  post-reform  sample. 

Hypothesis  Test  #2  -  Development  Phase 

The  first  hypothesis  tested  to  determine  if  the  mean  final  cost  overrun  percentage 
(FO%)  of  pre-reform  development  phase  contracts  (1  Jan  94-31  Dec  97)  is  significantly 
different  than  the  mean  FO%  of  post- reform  development  phase  contracts  (1  Jan  98  - 
31  Dec  01). 
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Normality  and  Equal  Variance  Test  Results 

The  Shapiro- Wilk  test  conducted  on  the  pre-reform  sample  produced  a  test 
statistic  value  of  .8833  and  a  p- value  of  .0010.  These  values  indicate  a  non-normal 
distribution  for  the  pre-reform  sample.  The  Shapiro- Wilk  test  conducted  on  the  post- 
reform  sample  produced  a  test  statistic  value  of  .8506  and  a  p- value  of  less  than  .0001. 
These  values  indicate  a  non-normal  distribution  for  the  post-reform  sample.  At  a 
significance  level  of  .05,  both  results  indicate  both  samples  are  non- normal,  therefore 
more  emphasis  will  be  placed  on  the  Mann- Whitney  test  results  over  the  two -sample 
t-test. 

The  Levene  test  for  equal  variances  produced  a  test  statistic  value  of  12.5972  and 
a  p- value  of  .0007.  This  result  indicates  that  the  population  variances  are  not  equal  at  a 
significance  level  of  .05.  Therefore,  the  Welch  ANOVA  test  results  will  be  utilized  in  the 
analysis. 

Hypothesis  Test  Results 

The  results  of  the  hypothesis  test  are  represented  in  Table  15.  The  test  statistic 
values  and  associated  p- values  are  consistent.  At  a  significance  level  of  .05,  the  results 
indicate  that  the  null  hypothesis  is  not  rejected. 


Table  15.  Phase  One  -  Hypothesis  Test  #2  Results 


Development  Phase 

Mean  FO% 

Std  Error 

Pre-Reform 

11.0208 

2.134 

Post-Reform 

7.0911 

1.93 

Test  Results 

Test  Statistic  (Ws) 
P-value 

Two-Sarrmle  t 

-1.366 

0.176 
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The  Mann- Whitney  results  indicate  that  the  two  sample  distributions  do  not  differ 
in  location  and  the  two-sample  t-test  indicates  the  mean  FO%  for  each  sample  is  not 
significantly  different.  Both  results  suggest  that  the  mean  FO%  of  the  pre-reform  sample 
is  equal  to  the  mean  FO%  of  the  post-reform  sample. 

Hypothesis  Test  #3  -  Production  Phase 

The  third  hypothesis  tested  to  determine  if  the  mean  final  cost  overrun  percentage 
(FO%)  of  pre-reform  production  phase  contracts  (1  Jan  94-31  Dec  97)  is  significantly 
different  than  the  mean  FO%  of  post- reform  production  phase  contracts  (1  Jan  98  - 
31  Dec  01). 

Normality  and  Equal  Variance  Test  Results 

The  Shapiro- Wilk  test  conducted  on  the  pre-reform  sample  produced  a  test 
statistic  value  of  .7144  and  a  p- value  of  less  than  .0001.  These  values  indicate  a  non¬ 
normal  distribution  for  the  pre-reform  sample.  The  Shapiro- Wilk  test  conducted  on  the 
post-reform  sample  produced  a  test  statistic  value  of  .6022  and  a  p-value  of  less  than 
.0001.  These  values  indicate  a  non-normal  distribution  for  the  post-reform  sample.  At  a 
significance  level  of  .05,  both  results  indicate  both  samples  are  non- normal;  therefore 
more  emphasis  will  be  placed  on  the  Mann- Whitney  test  results  over  the  two -sample 
t-test. 

The  Levene  test  for  equal  variances  produced  a  test  statistic  value  of  .5688  and  a 
p-value  of  .4527.  This  result  indicates  that  the  population  variances  are  equal  at  a 
significance  level  of  .05.  Therefore,  the  pooled  t  test  results  will  be  utilized  in  the 
analysis. 
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Hypothesis  Test  Results 

The  results  of  the  hypothesis  test  are  represented  in  Table  16.  The  test  statistic 
values  and  associated  p- values  are  consistent.  At  a  significance  level  of  .05,  the  results 
indicate  that  the  null  hypothesis  is  not  rejected. 


Table  16.  Phase  One  -  Hypothesis  Test  #3  Results 


Production  Phase 

Mean  FO% 

Std  Error 

Pre-Reform 

8.1880 

2.216 

Post-Reform 

6.7682 

2.968 

Test  Results 

Test  Statistic  (Ws) 
P-value 

Two-Sairmle  t 

-0.383 

0.7023 

The  Mann- Whitney  results  indicate  that  the  two  sample  distributions  do  not  differ 
in  location  and  the  two-sample  t-test  indicates  the  mean  FO%  for  each  sample  is  not 
significantly  different.  Both  results  suggest  that  the  mean  FO%  of  the  pre-reform  sample 
is  equal  to  the  mean  FO%  of  the  post-reform  sample  for  production  phase  contracts. 

Hypothesis  Test  #4  -  Cost  Plus  Contracts 

The  fourth  hypothesis  tested  to  determine  if  the  mean  final  cost  overrun 
percentage  (FO%)  of  cost-plus  pre-reform  contracts  (1  Jan  94-31  Dec  97)  is 
significantly  different  than  the  mean  FO%  of  cost-plus  post- reform  contracts  (1  Jan  98  - 
31  Dec  01). 

Normality  and  Equal  Variance  Test  Results 

The  Shapiro- Wilk  test  conducted  on  the  pre-reform  sample  produced  a  test 
statistic  value  of  .8454  and  a  p- value  of  less  than  .0001.  These  values  indicate  a  non¬ 
normal  distribution  for  the  pre-reform  sample.  The  Shapiro-Wilk  test  conducted  on  the 
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post-reform  sample  produced  a  test  statistic  value  of  .8461  and  a  p- value  of  less  than 


.0001.  These  values  indicate  a  non- normal  distribution  for  the  post- reform  sample.  At  a 
significance  level  of  .05,  both  results  indicate  both  samples  are  non- normal,  therefore 
more  emphasis  will  be  placed  on  the  Mann- Whitney  test  results  over  the  two-sample 
t-test. 


The  Levene  test  for  equal  variances  produced  a  test  statistic  value  of  7.2882  and  a 
p-value  of  .0082.  This  result  indicates  that  the  population  variances  are  not  equal  at  a 
significance  level  of  .05.  Therefore,  the  Welch  ANOVA  test  results  will  be  utilized  in  the 
analysis. 

Hypothesis  Test  Results 

The  results  of  the  hypothesis  test  are  represented  in  Table  17.  The  test  statistic 
values  and  associated  p- values  are  consistent.  At  a  significance  level  of  .05,  the  results 
indicate  that  the  null  hypothesis  is  not  rejected. 


Table  17.  Phase  One  -  Hypothesis  Test  #4  Results 


Cost  Plus  Contracts 

Mean  FO% 

Std  Error 

Pre-Reform 

5.1422 

1.739 

Post-Reform 

5.5138 

1.420 

Test  Results 

Test  Statistic  (Ws) 
P-value 

Two-SamDle  t 

0.165 

0.8689 

The  Mann- Whitney  results  indicate  that  the  two  sample  distributions  do  not  differ 
in  location  and  the  two- sample  t-test  indicates  the  mean  FO%  for  each  sample  is  not 
significantly  different.  Both  results  suggest  that  the  mean  FO%  of  the  pre-reform  sample 
is  equal  to  the  mean  FO%  of  the  post-reform  sample  for  cost  plus  contracts. 
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Hypothesis  Test  #5  -  Fixed  Price  Contracts 

The  fifth  hypothesis  tested  to  determine  if  the  mean  final  cost  overrun  percentage 
(FO%)  of  fixed  price  pre-reform  contracts  (1  Jan  94-31  Dec  97)  is  significantly 
different  than  the  mean  FO%  of  fixed  price  post-reform  contracts  (1  Jan  98-31  Dec  01). 

Normality  and  Equal  Variance  Test  Results 

The  Shapiro-Wilk  test  conducted  on  the  pre-reform  sample  produced  a  test 
statistic  value  of  .7921  and  a  p- value  of  less  than  .0001.  These  values  indicate  a  non¬ 
normal  distribution  for  the  pre-reform  sample.  The  Shapiro-Wilk  test  conducted  on  the 
post-reform  sample  produced  a  test  statistic  value  of  .6382  and  a  p-value  of  less  than 
.0001.  These  values  indicate  a  non- normal  distribution  for  the  post- reform  sample.  At  a 
significance  level  of  .05,  both  results  indicate  both  samples  are  non- normal,  therefore 
more  emphasis  will  be  placed  on  the  Mann- Whitney  test  results  over  the  two -sample 
t-test. 

The  Levene  test  for  equal  variances  produced  a  test  statistic  value  of  2.2736  and  a 
p-value  of  .1353.  This  result  indicates  that  the  population  variances  are  equal  at  a 
significance  level  of  .05.  Therefore,  the  pooled  t  test  results  will  be  utilized  in  the 
analysis. 

Hypothesis  Test  Results 

The  results  of  the  hypothesis  test  are  represented  in  Table  18.  The  test  statistic 
values  and  associated  p- values  are  consistent.  At  a  significance  level  of  .05,  the  results 
indicate  that  the  null  hypothesis  is  not  rejected. 
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Table  18.  Phase  One  -  Hypothesis  Test  #5  Results 


Fixed  Price  Contracts 

Mean  FO% 

Std  Error 

Pre-Reform 

8.6052 

2.156 

Post-Reform 

8.4414 

3.075 

Test  Results 

Test  Statistic  (Ws) 
P-value 

Two-SamDle  t 

-0.044 

0.9653 

The  Mann- Whitney  results  indicate  that  the  two  sample  distributions  do  not  differ 
in  location  and  the  two- sample  t-test  indicates  the  mean  FO%  for  each  sample  is  not 
significantly  different.  Both  results  suggest  that  the  mean  FO%  of  the  pre-reform  sample 
is  equal  to  the  mean  FO%  of  the  post-reform  sample  for  fixed  price  contracts. 

Sensitivity  Testing 

Prior  research  indicates  that  over  target  baseline  contracts  may  not  be  reliable  data 
points  for  the  hypothesis  testing.  Over  target  baseline  (OTB)  contracts  are  contracts 
where  the  Total  Allocated  Budget  (TAB)  is  greater  than  the  Contract  Budget  Base 
(CBB),  which  is  evidence  that  a  contract’s  performance  measurement  baseline  exceeds 
the  project  targets  and  therefore  is  unstable  (Gordon,  1996:3).  To  test  the  sensitivity  of 
the  Phase  One  results,  all  OTB  contracts  were  removed  from  the  samples  and  the 
hypothesis  tests  were  performed  on  the  reduced  samples.  Table  19  displays  the  new 
sample  sizes  by  removing  the  OTB  contracts.  The  number  of  OTB  contracts  removed  is 
in  parentheses. 

The  test  statistic  values  and  associated  p-values  are  found  in  Appendix  C.  All  test 
values  indicate  no  significant  differences  noted  when  removing  the  OTB  contracts.  In  all 
cases,  the  null  hypotheses  were  not  rejected  at  a  significance  level  of  .05. 
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Table  19.  Phase  One  Samples  (OTB  Removed) 


Group 

Pre-Reform 

Post-Reform 

(1  Jan  94  -  31  Dec  97) 

(1  Jan  98-31  Dec  01) 

All  DoD  contracts 

75  (34) 

75  (20) 

Proaram  Phase 

Production 

Development 

Contract  TvDe 

Cost  Plus 
Fixed  Price 

32  (8) 

34  (25) 

47  (13) 

22  (7) 

Christensen  and  Templin  discovered  in  their  research  that  transitional  contracts 
tend  to  experience  poor  cost  performance  (Christensen  and  Templin,  2002:114). 
Transitional  contracts  are  defined  as  those  contracts  that  start  before  the  treatment  date 
and  end  after  the  treatment  date.  To  test  for  the  sensitivity  of  transitional  contracts,  the 
two  samples  used  in  the  original  hypothesis  testing  are  separated  into  three  samples;  pre¬ 
reform,  transitional,  and  post-reform.  Pre-reform  is  the  same  as  previously  described  in 
Chapter  3.  Transitional  contracts  are  those  which  started  prior  to  31  December  97  and 
finished  before  31  December  01,  while  post- reform  contracts  are  those  that  started  after 
31  December  97  and  finished  before  31  December  01.  Table  20  displays  the  sample  size 
numbers  for  the  three  samples  for  each  respective  hypothesis  test.  The  samples  were 
tested  with  and  without  the  OTB  contracts  for  robustness. 

In  order  to  test  for  a  difference  in  the  three  samples  the  use  of  different  hypothesis 
testing  procedures  is  required.  The  parametric  test  employed  is  the  single- factor  analysis 
of  variance  (ANOVA).  The  ANOVA  test  utilizes  the  F-distribution  and  similar 
assumptions  apply  as  did  for  the  two -sample  t  test;  normally  distributed  and  equal 
variance  of  the  samples.  As  noted  in  all  two -sample  testing,  the  samples  are  not  normally 
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distributed,  but  the  ANOVA  test  is  conducted  and  reported  for  robustness.  The  non- 
parametric  test  used  is  the  extension  of  the  Mann- Whitney  and  Wilcoxon  test  for 
independent  samples;  the  KruskafWallis  test  (Gibbons,  1985:317).  Both  procedures  test 
to  see  if  any  of  the  means  are  different  (Devore,  2000:403-404;  Gibbons,  1985:317) 

The  resultant  test  statistic  and  p- values  are  in  Appendix  C.  The  results  indicate  no 
significant  difference  between  the  means  at  a  .05  significance  (alpha)  level. 


Table  20.  Phase  One  -  Transitional  Period  Sensitivity  Test  Samples 


Group 

Pre-Reform 

Transitional 

Post-Reform 

Contract  Start 

Prior  to  31  Dec  1997 

Prior  to  31  Dec  1997 

1  Jan  1998  or  later 

Contract  ComDlete 

Prior  to  31  Dec  1997 

1  Jan  1998  or  later 

1  Jan  1998  or  later 

All  DoD  contracts 

109 

86 

9 

Proaram  Phase 

Production 

Development 

61 

36 

4^  GO 

O  O 

4 

4 

Contract  Tvoe 

Cost  Plus 
Fixed  Price 

40 

59 

54 

24 

6 

5 

Phase  Two 

The  purpose  of  Phase  Two  was  to  analyze  cost  performance  data  over  time  to  see 
any  visible  indications  of  acquisition  reform  initiative  implementation  and  cost 
performance  change.  The  analysis  consisted  of  visually  analyzing  the  plotted  data  to  see 
any  indications  of  change.  The  second  part  of  the  analysis  is  to  apply  the  intervention 
concepts  discussed  in  Chapter  3  to  see  if  there  is  a  common  pattern  exhibited  between 
reform  implementation  and  cost  performance  behavior. 


64 


Trend  vs.  Timeline  Visual  Analysis 


The  first  step  in  this  portion  of  the  analysis  involved  setting  up  all  the  calculations 
explained  in  Chapter  3.  This  was  all  performed  utilizing  Microsoft  Excel.  The  next  step 
was  to  plot  the  values  calculated  from  equation  9  against  a  timeline  backdrop  of 
acquisition  reform  initiatives.  Figure  6  displays  the  graph  for  all  contracts  and 
Appendix  D  contains  graphs  for  all  DoD  contracts  broken  out  by  program  phase  and 
contract  type.  The  graph  in  Figure  6  displays  a  line  for  each  service  department  (Air 
Force,  Army,  Navy)  and  a  line  for  all  DoD,  which  includes  each  service  and  any 
contracts  that  are  not  service- specific  (e.g.  Ballistic  Missile  Defense). 


This  portion  of  the  analysis  highlights  some  noteworthy  information.  First,  each  service 
exhibits  common  cost  variance  trends  over  time.  Each  service  data  exhibits  common 
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peaks  and  valleys.  This  data  behavior  laid  against  the  timeline  provides  a  visual  tool  for 
assessment  using  the  intervention  analysis  concepts. 

Intervention  Analysis 

The  most  interesting  behavior  of  the  cost  variance  data  is  the  sudden  cost  variance 
drops  at  or  immediately  following  years  where  an  acquisition  reform  report  was  issued  or 
an  initiative  implemented.  Utilizing  the  intervention  concepts  used  in  forecasting,  this 
behavior  seems  to  follow  either  a  pulse  intervention  or  a  decayed-response  intervention. 
The  cost  variance  tends  to  immediately  improve  (better  performance)  following  an 
acquisition  reform  event.  Following  the  Carlucci  Initiatives  in  1982,  the  DoD-All  trend 
line  takes  a  dip  down  to  1.32%  in  1983.  By  1986,  the  cost  variance  value  is  back  to  6%. 
Following  the  Packard  Commission  in  1986,  the  cost  variance  value  is  2.86%  in  1987. 

By  1991,  the  cost  variance  is  up  to  22.93%,  but  on  the  heels  of  the  Defense  Management 
Report  falls  back  to  14.4%  in  1994.  In  1996,  two  years  after  the  Federal  Acquisition 
Streamlining  Act  of  1994  and  as  the  Clinger-Cohen  Act  of  1996  (FASA  II)  passes,  the 
DoD  cost  variance  value  is  down  to  3.04%,  and  has  not  surpassed  4%  since  (as  of  June 
2002). 

The  trend  pattern  exhibited  in  the  data  does  give  some  potential  to  developing  a 
rule  of  thumb.  Certainly,  something  happened  to  the  cost  performance  of  active  contracts 
during  the  different  acquisition  reform  events  laid  out  in  the  timeline.  Further  discussion 
of  this  relationship  will  occur  in  Chapter  5. 
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Phase  Three 


In  consideration  of  the  fact  that  Phase  One  determined  that  there  was  no 
significant  difference  between  the  pre-reform  and  post-reform  contracts  further  analysis 
was  conducted.  As  discussed  in  Chapter  3,  Phase  Three  follows  the  same  methodology 
as  Phase  One.  Phase  Three  uses  a  treatment  date  of  31  December  1994.  This  treatment 
date  was  selected  for  the  following  three  reasons: 

•  Deputy  Undersecretary  of  Defense  for  Acquisition  Reform  position 
created  in  June  1993; 

•  GAO  report  stating  DoD  had  complied  with  Title  V  of  the  FASA  by  Fiscal 
Year  1995  (Oct  1994);  and 

•  Active  contract  cost  variance  dip  discovered  in  Phase  Two  analysis. 

Like  Phase  One,  the  pre-reform  sample  will  consist  of  the  four  years  before  the  treatment 
date  and  the  post-reform  period  will  consist  of  the  four  years  following  the  treatment 
date.  Table  21  provides  information  on  the  sample  sizes  for  each  portion  of  the  analysis. 


Table  21.  Phase  Three  Sample  Statistics 


Group 

Pre-Reform 

Post-Reform 

(1  Jan  91  -  31  Dec  94) 

(1  Jan  95  -  31  Dec  98) 

All  DoD  contracts 

156 

116 

Proaram  Phase 

Production 

Development 

97 

42 

52 

44 

Contract  Tvoe 

Cost  Plus 
Fixed  Price 

32 

114 

57 

46 
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Hypothesis  Test  #1  -  All  DoD 

The  first  hypothesis  tested  to  determine  if  the  mean  final  cost  overrun  percentage 
(FO%)  of  all  pre-reform  contracts  (1  Jan  91-31  Dec  94)  is  significantly  different  than 
the  mean  FO%  of  all  post-reform  contracts  (1  Jan  95-31  Dec  98). 

Normality  and  Equal  Variance  Test  Results 

The  Shapiro-Wilk  test  conducted  on  the  pre-reform  sample  produced  a  test 
statistic  value  of  .7623  and  a  p- value  of  .0000.  These  values  indicate  a  non- normal 
distribution  for  the  pre-reform  sample.  The  Shapiro-Wilk  test  conducted  on  the  post¬ 
reform  sample  produced  a  test  statistic  value  of  .8108  and  a  p-value  of  .0000.  These 
values  indicate  a  non-normal  distribution  for  the  post-reform  sample.  At  a  significance 
level  of  .05,  both  results  indicate  both  samples  are  non-normal;  therefore  more  emphasis 
will  be  placed  on  the  Mann- Whitney  test  results  over  the  two- sample  t-test. 

The  Levene  test  for  equal  variances  produced  a  test  statistic  value  of  1.0291  and  a 
p-value  of  .3113.  This  result  indicates  that  the  population  variances  are  equal  at  a 
significance  level  of  .05.  Therefore,  the  pooled  t  test  results  will  be  utilized  in  the 
analysis. 

Hypothesis  Test  Results 

The  results  of  the  hypothesis  test  are  represented  in  Table  22.  The  test  statistic 
values  and  associated  p- values  are  consistent.  At  a  significance  level  of  .05,  the  results 
indicate  that  the  null  hypothesis  is  not  rejected. 
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Table  22.  Phase  Three  Hypothesis  Test  #1  Results 


All  DoD  contracts 

Mean  FO% 

Std  Error 

Pre-Reform 

9.3375 

1.333 

Post-Reform 

7.4027 

1.546 

Test  Results 

Test  Statistic  (Ws) 
P-value 

Two-SamDle  t 

-0.948 

0.344 

The  Mann- Whitney  results  indicate  that  the  two  sample  distributions  do  not  differ 
in  location  and  the  two- sample  t-test  indicates  the  mean  FO%  for  each  sample  is  not 
significantly  different.  Both  results  suggest  that  the  mean  FO%  of  the  pre-reform  sample 
is  equal  to  the  mean  FO%  of  the  post-reform  sample. 

Hypothesis  Test  #2  -  Development  Phase 

The  first  hypothesis  tested  to  determine  if  the  mean  final  cost  overrun  percentage 
(FO%)  of  pre-reform  development  phase  contracts  (1  Jan  91-31  Dec  94)  is  significantly 
different  than  the  mean  FO%  of  post- reform  development  phase  contracts  (1  Jan  95  - 
31  Dec  98). 

Normality  and  Equal  Variance  Test  Results 

The  Shapiro-Wilk  test  conducted  on  the  pre-reform  sample  produced  a  test 
statistic  value  of  .7802  and  a  p- value  of  less  than  .0001.  These  values  indicate  a  non¬ 
normal  distribution  for  the  pre-reform  sample.  The  Shapiro-Wilk  test  conducted  on  the 
post-reform  sample  produced  a  test  statistic  value  of  .8834  and  a  p-value  of  .0002.  These 
values  indicate  a  non-normal  distribution  for  the  post-reform  sample.  At  a  significance 
level  of  .05,  both  results  indicate  both  samples  are  non-normal,  therefore  more  emphasis 
will  be  placed  on  the  Mann- Whitney  test  results  over  the  two-sample  t-test. 
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The  Levene  test  for  equal  variances  produced  a  test  statistic  value  of  .0212  and  a 
p- value  of  .8846.  This  result  indicates  that  the  population  variances  are  equal  at  a 
significance  level  of  .05.  Therefore,  the  pooled  t  test  results  will  be  utilized  in  the 
analysis. 

Hypothesis  Test  Results 

The  results  of  the  hypothesis  test  are  represented  in  Table  23.  The  test  statistic 
values  and  associated  p- values  are  consistent.  At  a  significance  level  of  .05,  the  results 
indicate  that  the  null  hypothesis  is  not  rejected. 


Table  23.  Phase  Three  -  Hypothesis  Test  #2  Results 


Development  Phase 

Mean  FO% 

Std  Error 

Pre-Reform 

12.0255 

2.460 

Post-Reform 

10.7675 

2.403 

Test  Results 

Test  Statistic  (Ws) 
P-value 

Two-SamDle  t 

-0.366 

0.7154 

The  Mann- Whitney  results  indicate  that  the  two  sample  distributions  do  not  differ 
in  location  and  the  two-sample  t-test  indicates  the  mean  FO%  for  each  sample  is  not 
significantly  different.  Both  results  suggest  that  the  mean  FO%  of  the  pre-reform  sample 
is  equal  to  the  mean  FO%  of  the  post-reform  sample. 

Hypothesis  Test  #3  -  Production  Phase 

The  third  hypothesis  tested  to  determine  if  the  mean  final  cost  overrun  percentage 
(FO%)  of  pre-reform  production  phase  contracts  (1  Jan  91-31  Dec  94)  is  significantly 
different  than  the  mean  FO%  of  post- reform  production  phase  contracts  (1  Jan  95  - 
31  Dec  98). 
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Normality  and  Equal  Variance  Test  Results 

The  Shapiro- Wilk  test  conducted  on  the  pre-reform  sample  produced  a  test 
statistic  value  of  .7315  and  a  p- value  of  .0000.  These  values  indicate  a  non-normal 
distribution  for  the  pre-reform  sample.  The  Shapiro- Wilk  test  conducted  on  the  post¬ 
reform  sample  produced  a  test  statistic  value  of  .6685  and  a  p- value  of  less  than  .0001. 
These  values  indicate  a  non-normal  distribution  for  the  post-reform  sample.  At  a 
significance  level  of  .05,  both  results  indicate  both  samples  are  non- normal,  therefore 
more  emphasis  will  be  placed  on  the  Mann- Whitney  test  results  over  the  two -sample 
t-test. 

The  Levene  test  for  equal  variances  produced  a  test  statistic  value  of  1.6458  and  a 
p-value  of  .2015.  This  result  indicates  that  the  population  variances  are  equal  at  a 
significance  level  of  .05.  Therefore,  the  pooled  t  test  results  will  be  utilized  in  the 
analysis. 

Hypothesis  Test  Results 

The  results  of  the  hypothesis  test  are  represented  in  Table  24.  The  test  statistic 
values  and  associated  p- values  are  consistent.  At  a  significance  level  of  .05,  the  results 
indicate  that  the  null  hypothesis  is  not  rejected. 


Table  24.  Phase  Three  -  Hypothesis  Test  #3  Results 


Production  Phase 

Mean  FO% 

Std  Error 

Pre-Reform 

8.9002 

1.842 

Post-Reform 

6.0459 

2.516 

Test  Results 

Test  Statistic  (Ws) 
P-value 

Mann-Whitnev 

3619 

0.264 

Two-Sairmle  t 

-0.915 

0.3615 
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The  Mann- Whitney  results  indicate  that  the  two  sample  distributions  do  not  differ 
in  location  and  the  two-sample  t-test  indicates  the  mean  FO%  for  each  sample  is  not 
significantly  different.  Both  results  suggest  that  the  mean  FO%  of  the  pre-reform  sample 
is  equal  to  the  mean  FO%  of  the  post-reform  sample  for  production  phase  contracts. 

Hypothesis  Test  #4  -  Cost  Plus  Contracts 

The  fourth  hypothesis  tested  to  determine  if  the  mean  final  cost  overrun 
percentage  (FO%)  of  cost-plus  pre-reform  contracts  (1  Jan  91-31  Dec  94)  is 
significantly  different  than  the  mean  FO%  of  cost-plus  post- reform  contracts  (1  Jan  95  - 
31  Dec  98). 

Normality  and  Equal  Variance  Test  Results 

The  Shapiro- Wilk  test  conducted  on  the  pre-reform  sample  produced  a  test 
statistic  value  of  .7146  and  a  p- value  of  less  than  .0001.  These  values  indicate  a  non¬ 
normal  distribution  for  the  pre-reform  sample.  The  Shapiro- Wilk  test  conducted  on  the 
post-reform  sample  produced  a  test  statistic  value  of  .8537  and  a  p-value  of  less  than 
.0001.  These  values  indicate  a  non-normal  distribution  for  the  post-reform  sample.  At  a 
significance  level  of  .05,  both  results  indicate  both  samples  are  non- normal,  therefore 
more  emphasis  will  be  placed  on  the  Mann- Whitney  test  results  over  the  two -sample 
t-test. 

The  Levene  test  for  equal  variances  produced  a  test  statistic  value  of  .1816  and  a 
p-value  of  .6710.  This  result  indicates  that  the  population  variances  are  equal  at  a 
significance  level  of  .05.  Therefore,  the  pooled  t  test  results  will  be  utilized  in  the 
analysis. 
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Hypothesis  Test  Results 

The  results  of  the  hypothesis  test  are  represented  in  Table  25.  The  test  statistic 
values  and  associated  p- values  are  consistent.  At  a  significance  level  of  .05,  the  results 
indicate  that  the  null  hypothesis  is  not  rejected. 


Table  25.  Phase  Three  -  Hypothesis  Test  #4  Results 


Cost  Plus  Contracts 

Mean  FO% 

Std  Error 

Pre-Reform 

8.4189 

2.498 

Post-Reform 

5.9602 

1.872 

Test  Results 

Test  Statistic  (Ws) 
P-value 

Two-Sairmle  t 

-0.788 

0.4331 

The  Mann-Whitney  results  indicate  that  the  two  sample  distributions  do  not  differ 
in  location  and  the  two- sample  t-test  indicates  the  mean  FO%  for  each  sample  is  not 
significantly  different.  Both  results  suggest  that  the  mean  FO%  of  the  pre-reform  sample 
is  equal  to  the  mean  FO%  of  the  post-reform  sample  for  cost  plus  contracts. 

Hypothesis  Test  #5  -  Fixed  Price  Contracts 

The  fifth  hypothesis  tested  to  determine  if  the  mean  final  cost  overrun  percentage 
(FO%)  of  fixed  price  pre-reform  contracts  (1  Jan  91-31  Dec  94)  is  significantly 
different  than  the  mean  FO%  of  fixed  price  post- reform  contracts  (1  Jan  95-31  Dec  98). 

Normality  and  Equal  Variance  Test  Results 

The  Shapiro- Wilk  test  conducted  on  the  pre-reform  sample  produced  a  test 
statistic  value  of  .7591  and  a  p- value  of  .0000.  These  values  indicate  a  non- normal 
distribution  for  the  pre-reform  sample.  The  Shapiro-Wilk  test  conducted  on  the  post¬ 
reform  sample  produced  a  test  statistic  value  of  .8021  and  a  p- value  of  less  than  .0001. 
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These  values  indicate  a  non-normal  distribution  for  the  post-reform  sample.  At  a 
significance  level  of  .05,  both  results  indicate  both  samples  are  non- normal,  therefore 
more  emphasis  will  be  placed  on  the  Mann- Whitney  test  results  over  the  two -sample 
t-test. 

The  Levene  test  for  equal  variances  produced  a  test  statistic  value  of  1.9836  and  a 
p-value  of  .1610.  This  result  indicates  that  the  population  variances  are  equal  at  a 
significance  level  of  .05.  Therefore,  the  pooled  t  test  results  will  be  utilized  in  the 
analysis. 

Hypothesis  Test  Results 

The  results  of  the  hypothesis  test  are  represented  in  Table  26.  The  test  statistic 
values  and  associated  p- values  are  consistent.  At  a  significance  level  of  .05,  the  results 
indicate  that  the  null  hypothesis  is  not  rejected. 


Table  26.  Phase  Three  -  Hypothesis  Test  #5  Results 


Fixed  Price  Contracts 

Mean  FO% 

Std  Error 

Pre-Reform 

10.5542 

1.638 

Post-Reform 

6.9447 

2.579 

Test  Results 

Test  Statistic  (Ws) 
P-value 

Two-Sarrmle  t 

-1.181 

0.2392 

The  Mann- Whitney  results  indicate  that  the  two  sample  distributions  do  not  differ 
in  location  and  the  two- sample  t-test  indicates  the  mean  FO%  for  each  sample  is  not 
significantly  different.  Both  results  suggest  that  the  mean  FO%  of  the  pre-reform  sample 
is  equal  to  the  mean  FO%  of  the  post-reform  sample  for  fixed  price  contracts. 
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Sensitivity  Testing 


As  performed  in  Phase  One,  the  same  sensitivity  testing  for  the  effects  of  OTB 
and  transitional  contracts  was  conducted.  Table  27  provides  the  new  sample  sizes  for 
OTB  contract  removal  in  each  respective  category  of  hypothesis  testing  and  Table  28 
provides  new  sample  sizes  for  the  breakout  of  transitional  from  the  post-reform  samples 
in  each  respective  category.  The  test  results  are  contained  in  Appendix  C.  Like  Phase 
One,  the  results  were  insensitive  to  any  effects  from  OTB  contracts,  in  that  no 
significance  differences  were  found.  The  results  for  testing  for  effects  of  transitional 
contracts  by  using  the  ANOVA  and  Kruskal-Wallis  tests  also  indicated  no  significant 
differences  in  the  three  sample  means  at  a  .05  significance  level. 


Table  27.  Phase  Three  Samples  (OTB  Removed) 


Group 

Pre-Reform 

Post-Reform 

(1  Jan  91  -  31  Dec  94) 

(1  Jan  95-31  Dec  98) 

Ail  DoD  contracts 

118  (38) 

91  (25) 

Proaram  Phase 

Production 

Development 

79  (18) 

27  (15) 

35  (17) 

36  (8) 

Contract  Tvoe 

Cost  Plus 
Fixed  Price 

23  (9) 

87  (27) 

50  (7) 

29  (17) 
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Table  28.  Phase  Three  -  Transitional  Period  Sensitivity  Test  Samples 


Group 

Pre-Reform 

Transitional 

Post-Reform 

Contract  Start 

Prior  to  31  Dec  1994 

Prior  to  31  Dec  1994 

1  Jan  1995  or  later 

Contract  ComDlete 

Prior  to  31  Dec  1994 

1  Jan  1995  or  later 

1  Jan  1995  or  later 

All  DoD  contracts 

156 

106 

10 

Proaram  Phase 

Production 

Development 

97 

42 

47 

41 

5 

3 

Contract  Tvoe 

Cost  Plus 
Fixed  Price 

32 

114 

O  CM 
LO  'sj- 

7 

2 

Chapter  Summary 

Phase  One  tested  to  find  if  any  differences  between  pre-reform  and  post-reform 
contract  cost  performance  existed.  With  a  reform  treatment  date  of  31  December  1997, 
all  five  hypotheses  test  indicate  that  no  difference  exists.  During  the  sensitivity  testing 
for  effects  of  over  target  baseline  and  transitional  contracts,  the  results  were  consistent; 
no  differences  found.  These  results  indicate  no  significant  difference  between  pre-reform 
and  post- reform  contract  cost  performance  with  a  treatment  date  of  31  December  1997. 

Phase  Two  analysis  provided  a  time  series  plot  of  contract  cost  performance  over 
the  span  of  the  Defense  Acquisition  Executive  Summary  database  timeframe.  When 
plotted  against  major  acquisition  reform  initiatives,  contract  cost  performance  appears  to 
change  at  the  time  or  immediately  following  each  respective  acquisition  reform  report  or 
initiative.  Applying  some  forecasting  tools  to  describe  and  possibly  model  the  change, 
the  impact  appears  to  be  a  decayed  response,  where  the  change  in  cost  performance 
decays  over  time  back  to  some  steady  state.  This  behavior  is  found  to  be  common  in  both 
the  development  and  production  phases  and  for  fixed  price  and  cost  plus  contracts. 
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Phase  Three  tested  the  same  hypotheses  in  Phase  One,  but  with  a  new  treatment 
date  of  31  December  1994.  The  evidence  of  no  difference  noted  in  Phase  One,  some 
organizational  changes  within  the  DoD,  and  the  cost  performance  behavior  identified  in 
Phase  Two  were  the  genesis  for  the  new  treatment  date.  The  Phase  Three  results  indicate 
no  significant  difference  between  pre-reform  and  post-reform  contract  cost  performance. 
The  results  of  sensitivity  testing  for  effects  of  OTB  and  transitional  contracts  also 
indicated  no  significant  differences  between  the  sample  means. 

This  concludes  the  results  for  both  sets  of  hypothesis  tests  and  the  time  series  plot 
visual  analysis.  The  statistical  test  results  and  p- values  not  shown  in  this  chapter  are  in 
Appendix  C  and  additional  time  series  plots  are  available  in  Appendix  D.  Chapter  5  will 
draw  conclusions  on  these  results  and  discuss  recommendations  for  future  research  and 
study. 
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V.  Conclusion 


Review  of  Research  Objectives 

This  study  started  with  two  main  objectives.  First,  to  determine  if  a  mapping 
existed  between  acquisition  reform  implementation  and  a  change  in  contract  cost 
performance.  Second,  if  a  mapping  existed,  to  develop  a  rule  of  thumb  to  model  this 
behavior.  The  use  of  hypothesis  tests  assessed  any  statistical  indications  of  differences  in 
final  mean  contract  cost  overrun  percentages  of  pre-reform  and  post-reform  contracts  in 
the  Phase  One  and  Phase  Three  analyses.  The  hypotheses  analyzed  all  Defense 
Department  contracts  and  also  analyzed  breakouts  of  program  phase  and  contract  type  to 
discern  any  differences.  The  Phase  Two  analysis  involved  plotting  the  contract  cost 
variance  data  over  time  to  see  any  trends  or  changes  in  relation  to  acquisition  reform 
events.  The  results  of  all  three  phases  meet  the  objectives  stated  above. 

Discussion  of  Results 

Phase  One  results  indicated  no  statistically  significant  difference  in  the  mean  final 
cost  overrun  percentage  of  pre-reform  contracts  and  post-reform  contracts,  utilizing 
31  December  1997  as  the  treatment  date.  The  results  were  consistent  across  the  program 
phases  and  contract  types. 

Phase  Two  produced  interesting  results.  In  looking  at  the  time  plots  of  the 
contract  cost  variance  percentages,  there  are  noticeable  performance  changes.  These 
changes  in  performance  coincide  with  acquisition  reform  events.  The  consistency  of 
these  occurrences  gives  credence  to  the  existence  of  a  relationship  between  acquisition 
reform  events  and  active  contract  cost  performance.  However,  these  changes  are  not 
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statistically  quantified.  The  changes  in  cost  variance  percentage  are  based  on  a  visual 
determination  of  the  cost  variance  behavior  in  relationship  to  the  acquisition  reform 
timeline. 

Phase  Three  was  an  attempt  to  reestablish  a  treatment  date  with  information 
obtained  from  Phase  Two  as  well  as  information  of  internal  DoD  activities  toward 
acquisition  reform  implementation.  Using  a  treatment  date  of  31  December  1994,  the 
same  hypotheses  tested  in  Phase  One  were  tested  with  new  pre-reform  and  post- reform 
samples.  The  results  indicated  no  statistically  significant  difference  between  pre-reform 
and  post-reform  contract  cost  performance.  The  results  were  consistent  across  program 
phases  and  contract  types 

Conclusions 

The  results  of  Phases  One  and  Three  indicate  that  acquisition  reform  events  and 
initiatives  have  not  significantly  changed  the  final  contract  cost  overrun  percentage  of 
completed  contracts.  While  this  is  statistically  true,  this  is  not  an  indication  that 
acquisition  reform  is  not  achieving  intended  goals.  Any  acquisition  reform  effects  noted 
in  initial  program  cost  estimates  and  taken  into  account  prior  to  initiating  contracts  would 
not  be  reflected  in  these  results.  The  only  acquisition  reform  impacts  noted  here  would 
be  impacts  to  the  way  contracts  are  controlled  and  managed  to  completion.  The 
statistical  tests  indicate  that  cost  performance,  as  reflected  through  earned  value,  is  no 
different  for  post-reform  contracts  as  compared  to  pre-reform  contracts.  These 
conclusions  are  also  based  on  the  selection  of  31  December  1997  and  31  December  1994 
as  treatment  dates  for  Phase  One  and  Phase  Three,  respectively. 
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While  the  analysis  of  completed  contracts  indicates  no  change  in  performance,  the 
Phase  Two  results  tell  a  different  story.  Phase  Two  analyzed  active  contracts,  regardless 
of  percent  complete.  The  plots  indicate  that  active  contract  cost  performance  changes  as 
a  result  of  acquisition  reform  events.  There  appears  to  be  a  relationship  between 
acquisition  reform  events  and  an  immediate  change  in  cost  performance.  However, 
without  an  in-depth  analysis  of  each  respective  contract,  this  study  cannot  provide 
answers  as  to  why  such  changes  occurred  and  why  they  appear  to  be  short-  lived. 

The  results  of  this  study  are  consistent  with  past  empirical  studies  which 
analyzed  contract  cost  performance  following  acquisition  reform  events.  Christensen  and 
Templin  found  that  post  A- 12  contracts  performed  better  than  pre  A- 12  contracts.  The 
difference  in  treatment  dates  and  methodology  employed  account  for  the  different  results. 
Christensen  and  Templin  utilized  31  December  1991  as  the  treatment  date,  while  this 
study  used  dates  3  and  6  years  later  in  this  study.  The  results  of  this  study  indicate  that 
post  A- 12  (31  December  1991),  completed  contract  cost  performance  has  not  changed, 
for  better  or  worse. 

The  results  are  also  consistent  with  the  Blueprint  for  Action  report  issued  in 
February  2001.  Any  failure  to  succeed  and  meet  the  goals  of  acquisition  reform  is  not 
from  a  lack  of  good  ideas.  For  decades  the  same  issues,  problems  and  solutions  have 
been  exposed  and  offered  by  numerous  commissions  and  panels  (Druyun,  2001:41).  The 
report  points  out  that  while  “progress  has  been  made  on  some  fronts,  a  lack  of  top-level 
priority  has  limited  the  progress  made  to  date”  (Druyun,  2001:41).  The  push  for 
transformation  and  the  hard-hitting  attitude  of  the  Defense  Department  leadership  is 
certainly  encouraging  and  actions  such  as  canceling  the  DoD  5000  series  are  indicative  of 
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its  importance.  However,  unless  this  “change”  movement  is  institutionalized  by 
government  acquisition  and  contractor  personnel,  history  will  repeat  itself. 

Recommendations  for  Future  Research 

There  are  three  areas  of  future  research  that  should  be  considered.  First,  the 
results  of  Phase  Two  should  be  explored  in  more  detail.  Study  of  a  few  contracts  that 
experienced  a  change  in  contract  cost  performance  during  times  of  acquisition  reform 
events  may  uncover  what  caused  these  changes.  The  question  of  “Are  the  changes  from 
chance  occurrences,  or  earned  value  and  contract  rebaselining  activities,  or  by  utilizing 
smart  business  practices  and  sound  management?”  needs  to  be  answered.  Lessons 
learned  from  this  analysis  could  be  very  beneficial  to  the  acquisition  community  for 
incorporation  into  other  programs  and  contracts. 

Pilot  programs  are  often  mentioned  when  discussing  acquisition  reform  success 
stories.  When  analyzing  the  contracts  in  the  DAES  database  associated  with  the 
acquisition  pilot  programs,  the  cost  performance  is  favorable  compared  to  the  average.  It 
may  be  worthwhile  to  study  and  find  the  underlying  cause  of  these  contracts’  better  than 
average  cost  performance  to  see  if  there  are  lessons  learned  to  pass  on  to  other  programs 
as  well.  The  results  of  this  study  could  also  provide  the  Department  of  Defense  with 
further  evidence  that  pilot  programs  are  successful  and  should  be  expanded. 

Finally,  an  analysis  of  program  cost  growth  using  Selected  Acquisition  Report 
(SAR)  data  should  be  conducted.  This  would  provide  a  comparison  to  the  conclusions  of 
the  Drezner,  et  al.  study  in  1993,  to  find  if  acquisition  reform  of  the  1990’s  has  truly 
affected  the  cost  growth  of  DoD  programs.  The  SAR  analysis  would  uncover  cost 


81 


growth  or  decline  not  reflected  or  evidenced  in  the  DAES  database,  if  it  exists.  This  type 
of  analysis,  coupled  with  this  study,  would  provide  a  more  complete  answer  to  whether  or 
not  acquisition  reform  activities  in  the  1990’s  has  affected  program  cost  growth. 
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Appendix  A.  DSMC  Earned  Value  Management  Gold  Card 


Defense  Systems  Management  College 
Earned  Value  Management  Gold  Card 


EAC 

TAB 

BAC 


VARIANCES  (Favorable  is  positive,  Unfavorable  is  negative) 


•  Cost  Variance 

•  Schedule  Variance 

•  Variance  at  Completion 


CV  =  BCWP-ACWP 
S  V  =  BCWP-BCWS 
VAC  =  BAC -EAC 


cv%=  cv/bcwp 
SV%=  Sv/  BCWS 


PERFORMANCE  INDICES 

•  Cost  Efficiency 

•  Schedule  Efficiency 


(Favorable  is  >  1.0,  Unfavorable  is  <  1.0) 
CPI  =  bcwp/acwp 

sp|  _  BCWP /BCWS 


OVERALL  STATUS 


•  Percent  Complete 


BCWP  CUM 

BAC 


•  Percent  Spent 


ACWPCUM 

BAC 


TO  COMPLETE  PERFORMANCE  INDEX  (TCPI) 

=  WORK  REMAINING  =  BAC- BCWP  CUM 
TCP  (EAC)  COST  REMAINING  EAC -ACWPCUM 

ESTIMATE  AT  COMPLETION  (EAC  =  ACWP  +  Estimate  for  Remaining  Work) 


EAC  CPI  = 


BAC 
CPI  CUM 


■  EACcomposite  _  ACWP  CUM  + 


BAC -BCWP CUM 
(CPI  CUM)  •  (SPI  CUM) 


83 


TERMINOLOGY 

NCC  -Negotiated  Contract  Cost 
AUW  -Authorized  Unpriced  Work 
CBB  -  Contract  Budget  Base 
OTB  -OverTarget  Baseline 
TAB  -  Total  Allocated  Budget 
BAC  -  Budget  At  Completion 
PWB  -  Performance  Measurement 


MR  -  Management  Reserve 
UB  -  Undistributed  Budget 
CA  -  Control  Account 
WP  -Work  Package 
PP  -  Planning  Package 
BCWS-  Budgeted  Cost  for  Work  Scheduled 
BCWP-  Budgeted  Cost  for  Work  Performed 
ACWP-  Actual  Cost  of  Work  Performed 
EAC  -  Estimate  At  Completion 


Contract  price  less  profit  /  fee 

Work  authorized  to  start,  not  yet  negotiated 

Sum  of  NCC  and  AUW 

Sum  of  CBB  and  recognized  overrun 

Sum  of  all  contract  budgets  -  NCC,  CBB  or  OTB  ( includes  MR ) 

Cumulative  BCWS  -  total  end  point  of  PMB  ( excludes  MR ) 

Contract  time-phased,  budgeted  work  plan  ( excludes  MR ) 

Contractor  PM’s  Contingency  budget 

Broadly  defined  activities  not  yet  distributed  to  CAs 

Contractor  key  management  control  point  -  CWBS  element 

Near-term,  detaitplanned  activities  within  a  CA 

Far-term  CA  activities  not  yet  defined  into  detail  Work  Packages 

Value  of  work  scheduled  -  PLAN 

Value  of  work  completed  -  EARNED  VALUE 

Cost  of  work  completed  -  ACTUAL  COSTS  INCURRED 

Estimate  of  total  contract  costs 


EVM  POLICY  (POD  5000.2-R) 

ALTERNATIVE  E V  MANAGEMENT  APPLICATIONS 
LEVEL  f.  EVMS  Industry  Standards  Management  Application 
Contractor  management  system  certified  as  meeting  Industry  Standards 

•  Required  for  norr-FFP  contract  exceeding  $73M  RDT&E  or  $3f  5M  in  procurement  (CYOO$). 

•  PM  may  apply  to  contracts  below-threshold  —consider  benefits,  risk  and  criticality. 

•  Contractor  must  establish,  maintain,  and  use  a  system  that  meets  the  the  32  Industry  Standards. 

•  Cost  Performance  Report  (CPR)  delivered  as  aCDRL  item. 

•  5  Formats  (WBS,  Organization,  Baseline,  Staffing,  and  Explanations) 

LEVEL  2.  C/SSR  Management  Application 

Contractor  Management  system  not  certified 

•  Required  for  non-FFPcorrtract  exceeding  $6.3M  (CYOO$)  and  12  months  in  length. 

•’Reasonably  objective’  EV  methods  acceptable,  traceability  at  higher  level  (CA  vs  WF^ 

•The  CPR  or  the  Cost/Schedule  Status  Report  (C/SSR)  delivered  as  a  CDRL  item. 

EVM  Home  Page — http://wvwv.  acq  .osd.mil/pml 
DSMC  EV  EMail  Address  -EVM@DSMCDSM.MIL 

DSMCEV  Phone  No. -(703)  805-2848/2968  (DSN  655)  June  2000 
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Appendix  B.  Normal  Probability  Plots 


Phase  One  -  All  DoD  Contracts 


Phase  One  -  Development  Phase  Contracts 


DoD  Development  Overrun  (Phase  1)%  Postreform  Normality  Tests 

i  y  r3~ 
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Phase  One  -  Production  Phase  Contracts 


Phase  One  -  Cost  Plus  Type  Contracts 


DoD  Cost  Plus  Cost  Overrun  (Phase  1)%  Postreform  Normality  Tests 

1 

i 

5  as  a  a  is  is  a  ■  is 

i  iii  i  iii  i 

- 1 - 1 - 1 - 1 - 1 - 1 

•L)  -  °  -*■  M  « 

Normal  Quantile  Plot 
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Phase  One  -  Fixed  Price  Type  Contracts 


Phase  Three  - All  DoD  Contracts 
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Phase  Three  -  Development  Phase  Contracts 


Phase  Three  -  Production  Phase  Contracts 
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Phase  Three  -  Cost  Plus  Type  Contracts 


Phase  Three  -  Fixed  Price  Type  Contracts 
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Appendix  C.  Hypothesis  Sensitivity  Results 


Phase  One  Hypothesis  Test  Results  (No  OTB) 


All  DoD  Contracts 

Mean  FO% 

Std  Error 

Pre-reform 

8.0033 

1.770 

Post-Reform 

6.1755 

1.807 

Test  Results 

Mann-Whitnev  U 

Two-Sample  t 

Levene 

Test  Statistic  (Ws) 

5159.5 

-0.723 

2.3354 

P-value 

0.5151 

0.4711 

0.1286 

Development  Contracts 

Pre-reform 

Post-Reform 

Mean  FO% 

14.2662 

6.7387 

Std  Error 

2.755 

2.328 

Test  Results 

Mann-Whitnev  U 

Two-Sample  t 

Levene 

Test  Statistic  (Ws) 

853 

1.8639 

18.3603 

P-value 

0.1772 

0.0718 

<.0001 

Production  Contracts 

Pre-reform 

Post-Reform 

Mean  FO% 

6.3242 

7.1845 

Std  Error 

2.929 

3.541 

Test  Results 

Mann-Whitnev  U 

Two-Sample  t 

Levene 

Test  Statistic  (Ws) 

767 

0.187 

1.3183 

P-value 

0.2894 

0.8521 

0.2553 

Cost  Plus  Contracts 

Pre-reform 

Post-Reform 

Mean  FO% 

5.2030 

5.1440 

Std  Error 

2.102 

1.734 

Test  Results 

Mann-Whitnev  U 

Two-Sample  t 

Levene 

Test  Statistic  (Ws) 

1148 

0.0193 

8.2747 

P-value 

0.1891 

0.9847 

0.0042 

Fixed  Price  Contracts 

Pre-reform 

Post-Reform 

Mean  FO% 

9.6022 

9.4828 

Std  Error 

3.360 

4.177 

Test  Results 

Mann-Whitnev  U 

Two-Sample  t 

Levene 

Test  Statistic  (Ws) 

560 

-0.022 

0.8982 

P-value 

0.2646 

0.9823 

0.3475 
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Phase  One  Hypothesis  Test  Results  ( Pre/Trans/Post , 


|  All  DoD  Contracts  | 

Pre-Reform 

Transitional 

Post-Reform 

Mean  FO% 

8.2522 

6.2733 

6.0043 

Sample  Size 

109 

86 

9 

Test  Results 

Kruskal-Wallis 

ANOVA 

Levene 

Test  Statistic 

1.2287 

0.4771 

1 .4847 

P-value 

0.541 

0.6213 

0.2291 

|  Development  Contracts  j 

Pre-Reform 

Transitional 

Post-Reform 

Mean  FO% 

11.0208 

7.0036 

7.9658 

Sample  Size 

36 

40 

4 

Test  Results 

Kruskal-Wallis 

ANOVA 

Levene 

Test  Statistic 

0.6562 

0.7901 

6.3674 

P-value 

0.7203 

0.4792 

0.0028 

|  Production  Contracts  f 

Pre-Reform 

Transitional 

Post-Reform 

Mean  FO% 

8.1880 

6.9401 

5.4793 

Sample  Size 

61 

30 

4 

Test  Results 

Kruskal-Wallis 

ANOVA 

Levene 

Test  Statistic 

3.165 

0.0851 

0.3518 

P-value 

0.2055 

0.9184 

0.7044 

|  Cost  Plus  Contracts  j 

Pre-Reform 

Transitional 

Post-Reform 

Mean  FO% 

5.1422 

5.6923 

3.9067 

Sample  Size 

40 

54 

6 

Test  Results 

Kruskal-Wallis 

ANOVA 

Levene 

Test  Statistic 

2.0025 

0.481 

4.3012 

P-value 

0.3674 

0.624 

0.0162 

|  Fixed  Price  Contracts  j 

Pre-Reform 

Transitional 

Post-Reform 

Mean  FO% 

8.6052 

8.6842 

7.2758 

Sample  Size 

59 

24 

5 

Test  Results 

Kruskal-Wallis 

ANOVA 

Levene 

Test  Statistic 

2.6915 

0.0157 

1.3983 

P-value 

0.2603 

0.9844 

0.2526 

91 


Phase  One  Hypothesis  Test  Results  (No  OTB  &  Pre/T rans/Post. 


92 


93 


Phase  Three  Hypothesis  Test  Results  (Pre/T rans/Post. 


94 


Phase  Three  Hypothesis  Test  Results  (No  OTB  &  Pre/Trans/Post. 


95 


Appendix  D.  Phase  Two  Time  Series  Plots 


Development  Phase  Contracts 


DoD  Development  Phase 


Production  Phase  Contracts 


96 


97 


Appendix  E.  DAES  Data 


Phase  One  Data 


Date 


1/25/1994 


1/25/1994 


1/25/1994 


3/25/1994 


425/1994 


425/1994 


425/1994 


5/25/1994 


5/25/1994 


5/25/1994 


7/25/1994 


7/25/1994 


7/25/1994 


7/25/1994 


8/25/1994 


8/25/1994 


8/25/1994 


8/25/1994 


9/25/1994 


9/25/1994 


1 0/25/1 994 


1  C/25/1 994 


1  C/25/1 994 


11/25/1994 


12^25/1994 


1225/1994 


1225/1994 


1225/1994 


1225/1994 


1225/1994 


1/25/1995 


225/1995 


225/1995 

3/25/1995 


3/25/1995 


3/25/1995 


425/1995 


425/1995 


425/1995 


5/25/1995 


625/1995 


625/1995 

7/25/1995 


7/25/1995 


16.70 


2.90 


0.90 


12.50 


309.20 


27.60 


104.60 


11.90 


-11.70 


-13.40 


76.50 


-7.00 


-1.70 


188.70 


85.50 


36.50 


-25.80 


36.50 


16.00 


127.60 


88.10 


106.40 


-1.00 


5.20 


-2.50 


CO%  A3AP 


-2.46%  79.3 


1.30%  109.4 


66.87%  139 


0.27%  1004 


14.59%  509.8 


3.78% 


15.76% 


0.08% 


17.34% 


27.33% 


4.54% 


13.14% 


18.28% 


-3.88% 


-8.60% 


11.86% 


-5.75% 


-1.53% 


19.69% 


9.51% 


12.47% 


-5.52% 


5.02% 


10.70% 


14.73% 


16.01% 


7.75% 


-0.44% 


0.49% 


-3.20% 


458.8 


21.3 


1088.5 


84.6 


1440.6 


635.1 


900.9 


77 


289.9 


142.4 


721.6 


114.8 


109.6 


1147 


984.5 


329.1 


441.7 


764 


165.5 


993.6 


638.5 


1478.5 


224.4 


1067.7 


75.7 


11.47%  500.4 


1.31%  93.1 


-15.90%  16.4 


22.30  8.23%  293.1  270. 


2.70  0.96%  282.5  279. 


27.20  37.94%  98.9  71. 


194.60  40.70%  672.7  478.1 
30.30  2.21%  1400.9  1370. 


BCWP 


81  ' 


1 


442.1 


18.4 


1087.0 


72.1 


1131/ 


607. 


796.3 


65.1 


301.0 


155.0 


645.1 


121. 


111. 


958. 


899 


292.0 


467.9 


727.9 


149.9 


199.75%  Yes 


99.26% 


86.66%  No 


99.40%  No 


98.22%  No 


120.73%  Yes 


95.67%  No 


242.38%!  Yes 


125.1 7%  Yes 


124.45%  Yes 


94.25%  No 


97.67%Yes 


77.1 9%  Yes 


79.73%  No 


132.86%  Yes 


97.51%  No 


95.12%  No 


11 7.01%  Yes 


90.90%  No 


96.95%  No 


106.80%  Yes 


95.10%  No 


11 7.56%  Yes 


95.21% 


96.22%  No 


82.69%  No 


98.30%  No 


89.99%  No 


94.78%  No 


104.79%  Yes 


97.01%  No 


90.82%  No 


90.69%  No 
11 0.31%  Yes 


155.98%  Yes 


78.35%  No 


100.00%  No 


97.40%  No 


11 8.77%  Yes 


98.11%  No 


98.49%  No 


98.48%  No 
97.48%  No 


99.22%  No 


Phase 


Da/ 


Prod 


98 


8/25/1 995 


8/25/19951 


8/25/1995 


8/25/1995 


9/25/1995 


9/25/1995 


9/25/1995 


1  C/25/1 995 


1  C/25/1 995 


1  C/25/1 995 


1  C/25/1 995 


11/25/1995 


11/25/1995 


11/25/1995 


12/25/1995 


1/25/1996 

225/1996' 


3/25/1996 


4/25/1 996 


4/25/1 996 


4/25/1 996 


4/25/1 996 


5/25/1996 


525/1996 


525/1996 


625/1996 


625/1996 


625/1996 


625/1996 


7/25/1996 


7/25/1996 


7/25/1996 


8/25/1996 


925/1996 


1  C/25/1 996 


1  C/25/1 996 


11/25/1996 


11/25/1996 


1225/1996 


1225/1996 


1225/1996 


1/25/1997 


225/1997 


225/1997 


225/1997 


4/25/1 997 


525/1997 


525/1997 

525/1997' 


625/1997 


-0.10 


1.80 


2.70 


114.10 


29.10 


0.70 


-7.90 


4.40 


-2.60 


72.60 


2972.40 

26.10 


7.50 


12.10 


-11.50 


52.10 


33.00 


0.70 


-83.30 


42.50 


35.50 


-0.40 


4.50 


-1.20 


1.30 


-3.70 


474.60 


13.90 


0.00 


10.63%  122.8 


19.16%  70.9 


10.66%  183.8 


-5.60 


15.70 


-31.90 

1.00 


12.80 


-0.35% 


3.74% 


1.09% 


30.28% 


10.28% 


3.18% 


-9.23% 


-3.83% 


-1.82% 


46.60% 


57.36% 

31.11%" 


12.93% 


10.82% 


-1.80% 


8.86% 


35.68% 


0.17% 


-7.10% 


4.39% 


3.28% 


-0.70% 


-0.84% 


-2.88% 


7.93% 


-0.27% 


5.12% 


1.96% 


0.00% 


0.26% 


11.39% 


-3.18% 


14.05% 


0.28% 


-0.61% 


9.88% 


7.26% 


18.22% 


140.3 


228.4 


8154.7 

110 


65.5 


123.9 


628.5 


640.1 


125.5 


418.5 


1089.4 


1009.6 


1118.6 


56.8 


531.1 


40.4 


17.7 


1382.7 


9751.9 


722.8 


60.9 


76. 


92. 


1999. 


24. 


19709.1 


10962/ 


120/ 


111. 


1 


-1.13%  488.9  494. 


8.54%  199.6  183. 


-8.70 %  444.4  476. 
0.13%  767.8  766. 


11.55%  123.6  110. 


77.1 


102.7 


1936.4 


27.6 


19765.1 


10895.7 


132.3 


119.6 


172.6 


57. 


118. 


659. 


632 


95. 


424.1 


1099. 


965. 


953 


10514. 


735. 


68. 


97. 


77. 


92 


1911 


24.1 


10014. 


11779. 


127. 


104. 


149. 


80. 


520. 


189. 


5 

7 


122. 


168.69%  Yes 


89.34%  No 


130.07% 


97.87%  No 


99.31%  No 


140.23%  Yes 


93.69%  No 


97.54%  No 


11 9.80%  Yes 


90.91%  No 


95.86%  No 


99.05%  No 


97.04%  No 


91 .66%  No 


98.73%  No 


93.41%  No 
94.27%  No 


100.69%  Yes 


94.03%  No 


97.04%  No 


93.04%  No 


96.56%  No 


98.51%  No 


106.63%  Yes 


100.1 2%  Yes 


113.65% 


98.11%  No 


95. 13%  No 


94.98%  No 


95.91%  No 


97.55%  No 


88.24%  No 


96.41%Yes 


88.91%  No 


94.98%  No 


99.61%  No 


100.22%  No 


104.65%  Yes 


100.41%  No 


196.80%  Yes 


93.07%Yes 


94.21%  No 


106.60%  Yes 


97.59%  No 


11 0.10%  Yes 


94.99%  No 


97.15%  No 


92.90%  No 
108.58%  Yes 


90.45%  No 


Da/ 


Da/ 

Da/ 


Rod 


Da/ 


Rod 


Rod 


Da/ 


Da/ 


Rod 


Rod 


Da/ 


Rod 


Rod 


|  Training 


Da/ 


Rod 


Mix 


Da/ 


Da/ 


Da/ 


Da/ 


Mix 


Rod 


Mix 


Da/ 


Rod 


Da/ 


Rod 


Da/ 


Rod 


Rod 


Rod 


Rod 

Rod 


Da/ 


99 


7/25/1997 


7/25/1997 


8/25/1997 


825/1997 


825/1997 


825/1997 


826/1997 


9/25/1997 


1  C/25/1 997 


1  C/25/1 997 


11/25/1997 


11/25/1997 


11/25/1997 


11/25/1997 


12^25/1997 


1/25/98 

1/25/98" 


225/98 


325/98 


325/98 


425/98 


425/98 


525/98 


525/98 


525/98 


725/98 


725/98 


725/98 


825/98 


825/98 


925/98 


925/98 


925/98 


92598 


10/2598 


112598 


11/2598 


11/2598 


11/2598 


11/2598 


11/2598 


11/2598 


22599 


22599 


42599 


42599 


42599 


42599 

42599" 


52599 


87.60 


26.10 


0.80 


-3.10 


-19.60 


-3.80 


9.90 


225.50 


3.90 


-57.40 


49.70 


1.00 


2.70 


13.00 


9.60 


_ 434 

181.3 


-0.9 


18.3 


20.2 


6.9 


-19.6 


15.61% 


8.67% 


2.05% 


4.47% 


-18.63% 


4.09% 


5.26% 


20.11% 


2.20% 


-11.92% 


19.66% 


1.22% 


1.84% 


57.02% 


1.24% 


-0.34% 

28.47%" 


-2.72% 


30.65% 


17.26% 


4.04% 


-5.71% 


648.6 


327.1 


39.9 


66.3 


85.6 


89.2 


198.1 


1346.6 


180.8 


424 


302.5 


83.1 


149.7 


35.8 


786.4 


29.4 

818.2 


32.2 


78 


137.2 


177.5 


323.5 


4.78% 


3.91% 


13.03% 


14.94% 


-2.13% 


-0.61% 


-3.50% 


13.50% 


3.55% 


12.27% 


201.5 


919 


3267.6 


595.7 


-2.57% 


0.46% 


9.20% 


-2.45% 


8.40% 


-1.44% 


4.56% 


1.29% 


0.00% 

13.35%" 


6.23% 


561 


301 


39.1 


69/ 


105. 


93 


188. 


1121.1 


176.9 


481.4 


252.8 


82.1 


14 


22. 


776. 


29. 

636. 


33.1 


59.7 


117 


170.8 


343.1 


100.00%  No 


91 .46%  No 


99.49%  No 


99.14%  No 


95.81%  No 


98.10%  No 


93.17%  No 


105.72%  Yes 


77.49%  No 


96.16%  No 


98.90%  No 


95.58%  No 


93.63%  No 


86.69%  No 


11 4.83%  Yes 


99.66%  No 
100.63%  Yes 


79.38%  No 


98.03%Yes 


97.34%  No 


77.83%  No 


89.77%  No 


99.78%  No 


84.95%Yes 


79.30%  No 


74.64%  No 


95.17%  No 


98.92%  No 


86.24%  No 


97.43%  No 


94.93%  No 


86.46%  No 


93.30%  No 


95.95%Yes 


98.97%  No 


99.27%  No 


11 7.09%  Yes 


91 .15%  No 


101 .47%  No 


98.26%  No 


94.60%  No 


90.52%  No 


98.14%  No 


116.13% 


127.64%  Yes 


90.61%  No 


106.24%  Yes 


No 

No 


105.05%  Yes 


Yes 

CP 

Da/ 

Yes 

FP 

Prod 

Yes 

CP 

Da/ 

Yes 

CP 

Da/ 

Yes 

CP 

Da/ 

Yes 

CP 

Prod 

Yes 

CP 

FRod  i 

Yes 

CP 

Da/ 

Yes 

CP 

Da/ 

Yes 

CP 

Yes 

CP 

Da/ 

Yes 

CP 

Da/ 

Yes 

CP 

Da/ 

Yes 

Mix 

Prod 

Yes 

FP 

FRod  ! 

Yes 

Mix 

Da/ 

Yes 

CP 

Da/ 

Yes 

CP 

Hanning 

Yes 

FP 

Rod  | 

Yes 

FP 

Rod 

Yes 

CP 

Da/ 

Yes 

FP 

Rod  ! 

Yes 

FP 

Rod 

Yes 

FP 

[Rod 

Yes 

FP 

Rod 

Yes 

FP 

Rod 

Yes 

FP 

Rod  ! 

Yes 

CP 

Da/ 

Yes 

FP 

Rod  ! 

Yes 

CP 

PDRR  i 

Yes 

CP 

PDRR 

Yes 

CP 

PDFiR 

Yes 

CP 

PDRR 

Yes 

CP 

PDRR 

Yes 

CP 

Da/ 

100 


6/2599 


7/2599 


8/2599 


102599 


11/2599 


11/2599 


122599 


2/25/00 


2/2500 


22500 


32590 


325/00 


425/00 


525/00 


525/00 


625/00 
7/2500 " 


725/00 


725/00 


825/00 


825/00 


825/00 


925/00 


925/00 


925/00 


925/00 


1125/00 


11/25/00 


11/25/00 


125/01 


125/01 


125/01 


125/01 


125/01 


225/01 


225/01 


225/01 


225/01 


425/01 


425/01 


425/01 


525/01 


525/01 


725/01 


725/01 


725/01 


825/01 


825/01 
925/01 " 


925/01 


0.00% 


1.49% 


0.10% 


4.23% 


0.26% 


-0.50% 


9.44% 


-0.36% 


0.95% 


1.87% 


1.96% 


0.26% 


-3.71% 


3.05% 


18.28% 


3.02% 

-0.07%" 


38.37% 


11.64% 


6.70% 


-2.90% 


4.87% 


10.20% 


-5.80% 


1.26% 


7.89% 


0.00% 


-3.90% 


18.81% 


1.02% 


9.30% 


-0.54% 


0.54% 


4.85% 


6.12% 


-0.97% 


24.63% 


14.53% 


7.14% 


51.92% 


94.67% 


0.00% 


3.94% 


-0.75% 


1.03% 


-1.31% 


-1.92% 


14.74% 

-2.12%" 


12.36% 


276.3 


388.2 


102.7 


140.6 


114.3 


39.9 


267.7 


110.3 


913.2 


59.9 


265.8 


38.7 


199.8 


27 


438.7 


2114 

2618.2 


101.7 


280 


98.7 


730.9 


68.3 


28.1 


618.8 


56.1 


61.5 


40.8 


285.9 


1019.4 


3441.1 


738.1 


1518.1 


223.3 


1235.3 


175.2 


488.3 


50.6 


975.9 


213 


43.6 


32.9 


71.2 


163.5 


462.3 


597.1 


60.2 


137.8 


82.5 

3287.4 


362.8 


276. 


382. 


102. 


134. 


11 


40.1 


244. 


110. 


904. 


58. 


260. 


38. 


207. 


26. 


370. 


2052.1 

2620.1 


73. 


250. 


92. 


752. 


71. 


25. 


656. 


55/ 


5 


40. 


297. 


3406. 


675. 


1526. 


222.1 


1298.3 


165.1 


493.1 


40. 


852.1 


198. 


28. 


16. 


71. 


157. 


465. 


591 


61 


140. 


_ 71 

3358. 


322. 


11 0.74%  Yes 


98.51%  No 


96.16%  No 


90.29%  No 


96.28%  No 


98.28%  No 


99.96% 


84.76%  No 


99.64%  No 


100.17%  No 


89.93%  No 


85.02%  No 


96.83%  No 


98.13%  No 


91 .02%  No 


144.51%  Yes 
76.76%  Yes 


95.33%  No 


98.55%  No 


120.29%  Yes 


11 2.53%  Yes 


99.58%  No 


98.84%  No 


95.27%  No 


95.03%  No 


93.14%  No 


100.00%  No 


94.56%  No 


97.56%  No 


99.58%  No 


100.07%  Yes 


97.38%  No 


98.58%  No 


95.67%  No 


98.04%  No 


106.25%  Yes 


90.83%  No 


92.21%  No 


123.86%  Yes 


93.18%  No 


91 .35%  No 


83.47%  No 


94.42%  No 


91 .78%  No 


95.83%  No 


109.71%Yes 


90.24%  No 


97.29%  No 
95.30%  No 


92.18%  No 


Yes 

CP 

Da/ 

Yes 

CP 

Da/ 

Yes 

Mix 

Da/ 

Yes 

CP 

PDFIR 

Yes 

CP 

Da/ 

Yes 

CP 

Da/ 

Yes 

CP 

Da/ 

Yes 

CP 

Da/ 

Yes 

CP 

PDFIR  | 

Yes 

CP 

Yes 

CP 

Da/ 

NO 

FP 

Rod 

Yes 

FP 

Rod 

Yes 

CP 

Da/ 

Yes 

CP 

Da/ 

Yes 

CP 

Da/ 

Yes 

CP 

DEMVAL 

Yes 

CP 

Da/ 

Yes 

CP 

Da/ 

Yes 

CP 

Da/ 

Yes 

FP 

Rod 

Yes 

FP 

Rod  i 

NO 

CP 

Da/ 

Yes 

FP 

Rod  | 

Yes 

CP 

Yes 

CP 

Yes 

CP 

Da/ 

Yes 

FP 

Rod 

Yes 

FP 

Rod 

Yes 

CP 

Da/ 

Yes 

CP 

Da/ 

Yes 

CP 

Rod 

NO 

CP 

Rod  i 

Yes 

FP 

Rod 

Yes 

CP 

Da/ 

Yes 

FP 

Rod  i 

NO 

FP 

Rod  i 

Yes 

FP 

Rod 

Yes 

CP 

Rod 

Yes 

FP 

Rod 

Yes 

FP 

Rod 

Yes 

CP 

Da/ 

Yes 

CP 

Da/ 

Yes 

CP 

Closure 

NO 

FP 

Rod  ! 

Yes 

FP 

Rod  i 

Yes 

CP 

Da/ 

\smwsm 

Da/ 

Da/ 

Yes 

CP 

PDRR 

101 


9/25/01 

-6.7 

-3.27% 

198.5 

205.2 

209.6 

97.90% 

No 

Yes 

CP 

FYiasell 

1 0/25/01 

-65.6 

4.29% 

1465.1 

1530.7 

1602.6 

95.51% 

No 

NO 

FP 

F^nod 

11/25/01 

52.4 

6.02% 

922.9 

870.5 

672.8 

129.38% 

Yes 

CP 

Da/ 

11/25/01 

15.1 

12.73% 

133.7 

118.6 

138.8 

85.45% 

No 

Yes 

CP 

Da/ 

11/25/01 

0.3 

2.13% 

14.4 

14.1 

16.1 

87.58% 

No 

NO 

CP 

Da/ 

11/25/01 

2.7 

4.80% 

59 

56.3 

66.2 

85.05% 

No 

NO 

CP 

Da/ 

11/25/01 

89.6 

20.48% 

527.2 

437.6 

514.3 

85.09% 

No 

Yes 

CP 

Da/ 

11/25/01 

220.7 

92.75% 

No 

Yes 

Mix 

F^rod  ! 

1225/01 

-14.7 

-1.68% 

860.1 

874.8 

75.30% 

No 

Yes 

FP 

F^nod  i 

122501 

3.9 

4.87% 

84 

80.1 

96.86% 

No 

Yes 

Mix 

Da/ 

102 


Phase  Three  Data 


Date 

CV 

CO% 

ACWP 

BCWP 

BAC 

1/17/91 

4.1 

2.69% 

156.5 

152.4 

144.5 

1/25/91 

3.1 

3.52% 

91.1 

88 

28.2 

2/25/91 

16.8 

10.20% 

181.5 

164.7 

165.2 

2/25/91 

41.5 

6.27% 

703.1 

661.6 

666.1 

2/25/91 

1.3 

1.51% 

87.6 

86.3 

87.6 

3/25/91 

28.5 

18.45% 

183 

154.5 

156.4 

3/25/91 

-16.5 

-3.82% 

415.3 

431.8 

469.3 

3/25/91 

7.2 

4.45% 

169.1 

161.9 

168.7 

3/25/91 

-1.3 

-1.61% 

79.4 

80.7 

88.2 

4/25/91 

3.2 

3.81% 

87.1 

83.9 

89.1 

4/25/91 

0 

0.00% 

85.5 

85.5 

94.6 

4/25/91 

94.5 

25.27% 

468.5 

374 

382.9 

4/25/91 

6.1 

17.09% 

41 .8 

35.7 

37.5 

5/25/91 

-56.1 

-13.66% 

354.7 

410.8 

470 

5/25/91 

55.7 

7.21% 

828.7 

773 

670.8 

-1.31% 

45.1 

45.7 

47.9 

6/25/91 

6.5 

4.59% 

148.1 

141.6 

148.1 

6/25/91 

-10.4 

-12.67% 

71 .7 

82.1 

99.5 

7/10/91 

0 

0.00% 

27668 

27668 

10299 

7/25/91 

1.8 

1 .74% 

105.5 

103.7 

104.4 

7/25/91 

44.8 

15.07% 

342.1 

297.3 

303.8 

7/25/91 

32.2 

34.70% 

125 

92.8 

99.4 

7/25/91 

50.5 

18.94% 

317.1 

266.6 

283.5 

7/25/91 

14.7 

2.08% 

721.4 

706.7 

730.4 

7/25/91 

13.5 

7.40% 

195.9 

182.4 

194 

8/25/91 

14 

24.65% 

70.8 

56.8 

49.2 

8/25/91 

8.2 

9.11% 

98.2 

90 

90 

9/25/91 

7.9 

2.55% 

317.4 

309.5 

344.2 

9/25/91 

38.1 

16.49% 

269.1 

231 

306.2 

9/25/91 

23.1 

5.88% 

416.1 

393 

246.7 

9/25/91 

-3.8 

-3.97% 

91 .8 

95.6 

95.5 

10/25/91 

-15.8 

-2.88% 

532.3 

548.1 

559.9 

10/25/91 

2.5 

6.91% 

38.7 

36.2 

41.8 

10/25/91 

32.1 

4.01% 

832 

799.9 

828.1 

11/25/91 

13.6 

9.41% 

158.2 

144.6 

151.6 

11/25/91 

2.7 

2.60% 

106.4 

103.7 

108.8 

12/25/91 

5.9 

1 .46% 

410.5 

404.6 

409 

12/25/91 

0.4 

168.9 

12/25/91 

71.6 

189.3 

12/25/91 

35.2 

21.18% 

201.4 

166.2 

175.1 

1/25/92 

11.9 

18.62% 

75.8 

63.9 

66.1 

1/25/92 

9.5 

15.63% 

70.3 

60.8 

65.9 

1/25/92 

-5.7 

-7.30% 

72.4 

78.1 

79.2 

1/25/92 

52.9 

9.13% 

632.4 

579.5 

589.3 

1/25/92 

7.3 

1 .80% 

412.8 

405.5 

445.8 

105.47%  No 


31 2.06%  Yes 


99.70%  No 


99.32%  No 


98.52%  No 


98.79%  No 


92.01%  No 


95.97%  No 


91 .50%  No 


94.1 6%  No 


90.38%  No 


97.68%  No 


95.20%  No 


87.40%  No 


11 5.24%  Yes 


97.41%  No 
96.45%  No 


95.41%  Yes 


95.61%  No 


82.51%  No 


268.65%  Yes 


99.33%  No 


97.86%  No 


93.36%  No 


94.04%  No 


96.76%  No 


94.02%  No 


11 5.45%  Yes 


100.00%  No 


89.92%  Yes 


75.44%  No 


159.30%  Yes 


100.10%  No 


97.89%  No 


86.60%  No 


96.59%  No 


95.38%  No 


95.31%  No 


98.92%  No 


98.31%  No 
99.21%  No 


94.92%  No 


96.67%  No 


92.26%  No 


98.61%  No 


98.34%  No 


90.96%  No 


Phase 


DEV 


DEV 


DEV 


DEV 


DEV 


Prod 


Prod 


Prod 


Prod 


DEV/PROD 


DEV 


DEV 


Prod 


Devel/Prod 


DEV 


Prod 


Prod 


Prod 


Prod 


Prod 


DEV 


DEV 


DEV 


DEV 


Prod 


Prod 


Prod 


Prod 


Prod 


Prod 


DEV 


DEV 

Prod 


Prod 


DEV 


DEV 


Prod 


Prod 


QT 


103 


1/25/92 

-5.1 

2/25/92 

176.5 

2/25/92 

28.4 

2/25/92 

2.4 

2/25/92 

97.3 

3/25/92 

13.2 

3/25/92 

-3.1 

3/25/92 

72.2 

3/25/92 

14.9 

3/25/92 

5.2 

4/15/92 

54 

4/25/92 

311.6 

4/25/92 

75.2 

5/25/92 

1.4 

5/25/92 

18.5 

25.1 

EHSl 

-1.4 

5/25/92 

23 

5/25/92 

-31.6 

5/25/92 

15.2 

5/25/92 

57.7 

HSU 

6/25/92 

28.1 

6/25/92 

0.5 

7/25/92 

4.1 

7/25/92 

40.2 

8/25/92 

-12 

9/25/92 

40.1 

9/25/92 

-2 

10/25/92 

1 

10/25/92 

24.3 

10/25/92 

155.5 

10/25/92 

1.4 

11/1/92 

-9.3 

11/25/92 

14.7 

11/25/92 

4.4 

11/25/92 

18.8 

12/25/92 

2.9 

12/25/92 

-0.1 

1/22/93 

20.2 

1/25/93 

0.3 

1/25/93 

-2.9 

1/25/93 

-0.1 

1/25/93 

-12.2 

1/25/93 

-13.3 

1/25/93 

5.7 

1/25/93 

19.7 

2/25/93 

3.5 

-3.66% 


38.09% 


25.75% 


1.10% 


19.48% 


12.54% 


29.77% 


5.89% 


8.39% 


45.26% 


26.56% 


32.33% 


1.15% 


17.18% 


314.7 


268 


67.2 


173.3 


1484.8 


307.8 


123.6 


126.2 


11.84% 


-5.23% 


6.98% 


12.09% 


7.40% 


0.25% 


0.50% 


-3.20% 


-12.26% 


47.74% 


-1.69% 


0.86% 


62.47% 


8.73% 


1.22% 


-7.53% 


7.77% 


2.31% 


30.08% 


2.20% 


-0.56% 


33.06% 


0.18% 


-1.19% 


-0.04% 


-3.77% 


4.32% 


16.69% 


24.05% 

3.85%" 


3.38% 


139.4 


463.4 


110.3 


217.2 


499.4 


105.3 


875.5 


242.5 


253.1 


62 


119.3 


1173.2 


232.6 


122.2 


107.7 


408 


204.4 


822.9 


1216.2 


85.9 


124.1 


116.1 


117.9 


63.2 


1936.1 


116.4 


114.2 


203.8 


194.6 


81 .3 


135 


17.6 


81 .3 


165 


240.8 


222.6 


311.5 


140.3 


424.1 


114.5 


159.3 


384.8 


127.9 


917.8 


247.6 


264.5 


62.1 


126.8 


1182.2 


175.6 


118.5 


124.7 


194.2 


604.5 


217.9 


477.4 


123 


379.9 


203.9 


818.8 


1256.4 


97.9 


84 


118.1 


116.9 


38.9 


1780.6 


115 


123.5 


189.1 


190.2 


62.5 


132.1 


17.7 


61.1 


164.7 


243.7 


222.7 


323.7 


163.8 


132 


118 


92.3 

93.6" 


103.4 


201.5 


620.1 


261.7 


508.5 


143.5 


385.7 


224.8 


837.6 


1194.5 


126.8 


85.8 


137.7 


126.4 


41.3 


1529.7 


119.1 


130.5 


232.8 


194.9 


66.3 


142.9 


18.5 


67.7 


171 .4 


280.7 


236.3 


349.2 


171 .5 


137 


90.3 


_ 93 

98.9 


108.3 


99.36% 

No 

109.27% 

Yes 

96.33% 

No 

136.35% 

Yes 

129.78% 

Yes 

82.33% 

No 

95.39% 

No 

97.94% 

No 

95.69% 

No 

99.84% 

No 

94.09% 

No 

99.24% 

No 

132.46% 

Yes 

103.12% 

Yes 

86.37% 

No 

98.86% 

0 

93.01% 

0 

96.38% 

No 

97.48% 

No 

83.26% 

No 

93.88% 

No 

85.71% 

No 

98.50% 

No 

90.70% 

No 

97.76% 

No 

105.18% 

Yes 

77.21% 

No 

97.90% 

No 

85.77% 

No 

92.48% 

No 

94.19% 

No 

116.40% 

Yes 

96.56% 

No 

94.64% 

No 

81 .23% 

No 

97.59% 

No 

94.27% 

No 

92.44% 

No 

95.68% 

No 

90.25% 

Yes 

96.09% 

No 

86.82% 

No 

94.24% 

No 

92.70% 

No 

95.51% 

No 

96.35% 

No 

130.68% 

Yes 

99.25% 

Q 

0 

94.64% 

E 

0 

95.48% 

No 

CP  DEV 


FP  DEV 


FP  Prod 


FP 


FP 


FP  DEV 


CP  DEV 


FP  Prod 


FP  Prod 


FP  Prod 


FP  Prod 


FP  Prod 


CP  DEV 


CP  DEM/VAL 


CP  DEM/VAL 


Prod 

Prod 


FP  Prod 


FP  Prod 


CP  SDS 


FP 


CP 


FP  Prod 


CP  DEV 


CP  Prod 


FP  Prod 


FP  Prod 


FP  Prod 


Mix  Prod 


FP  Mix 


FP  DEV 


FP  DEV 


FP  Prod 


Mix  Prod 


FP  Prod 


FP  Prod 


FP  DEV 


FP  Prod 


FP  DEV 


CP  DEV 


CP  Prod 


Mix  Prod 


Mix  Prod 


Mix  Prod 


FP  Prod 


FP  Prod 


CP  DEV 


DEV 

Prod 


FP  Prod 


104 


2/25/93 

5.8 

3/25/93 

1.2 

3/25/93 

23.3 

3/25/93 

44.2 

3/25/93 

42.5 

3/25/93 

-5.8 

-4.4 

4/25/93 

4.6 

4/25/93 

12.5 

5/25/93 

26.9 

5/25/93 

117.8 

5/25/93 

21.9 

5/25/93 

-7.8 

6/25/93 

10.7 

6/25/93 

144.1 

■Sgll 

7/25/93 

19.3 

8/25/93 

50.6 

8/25/93 

32.5 

8/25/93 

-58.1 

9/17/93 

46.6 

9/25/93 

-1.3 

-2 

1  10/25/93 

75 

SSfl 

12/25/93 

23.8 

1/25/94 

-2 

1/25/94 

1.4 

1/25/94 

6.7 

3/25/94 

55.7 

4/25/94 

2.7 

4/25/94 

64.9 

4/25/94 

4.6 

5/25/94 

2.9 

5/25/94 

0.9 

7/25/94 

12.5 

7/25/94 

309.2 

7/25/94 

104.6 

7/25/94 

27.6 

8/25/94 

11.9 

8/25/94 

-11.7 

8/25/94 

-13.4 

8/25/94 

76.5 

9/25/94 

-7 

9/25/94 

-1.7 

10/25/94 

188.7 

10/25/94 

85.5 

10/25/94 

36.5 

3.68% 


2.20% 


25.55% 


19.91% 


8.13% 


-5.05% 


-12.83% 


6.63% 


3.10% 


70.05% 


145.07% 


11.26% 


-5.83% 


18.23% 


20.03% 


2.45% 


22.84% 


5.19% 


-11.33% 


21 .45% 


-0.85% 


5.36% 


-1.44% 


5.19% 


-2.46% 


1.30% 


3.86% 


66.87% 


0.27% 


14.59% 


4.53% 


15.76% 


0.08% 


17.34% 


27.33% 


13.14% 


4.54% 


18.28% 


-3.88% 


234.2 


1475.2 


609.9 


482.2 


79.3 


109.4 


180.1 


139 


1004 


509.8 


106.2 


21 .3 


1088.5 


84.6 


1440.6 


900.9 


635.1 


77 


289.9 


I  iiM  1  I 


11.86% 


-5.75% 


-1 .53% 


12.47% 


157.7 

167.9 

54.6 

57.6 

91.2 

95 

222 

232.4 

523 

296 

114.8 

120.8 

34.3 

34.3 

69.4 

72.3 

403.5 

442.7 

38.4 

38.4 

81.2 

84 

194.5 

199.8 

133.9 

141.6 

58.7 

61.6 

719.5 

722.8 

788.9 

796.7 

221.5 

194.1 

626 

731 

512.8 

540 

217.3 

271 

174.5 

219.1 

236.2 

208.6 

1400.2 

1061.3 

618.8 

610.7 

169 

119.3 

458.4 

290.3 

81.3 

40.7 

108 

108.8 

173.4 

200.1 

83.3 

83.8 

1001.3 

1019.4 

444.9 

368.5 

101.6 

106.2 

18.4 

14.7 

1087.6 

873.9 

72.1 

76.5 

1131.4 

1158.4 

796.3 

998.8 

607.5 

787 

65.1 

49 

301.6 

309.3 

155.8 

163.8 

645.1 

551.3 

121.8 

134 

111.3 

114.8 

958.3 

897.3 

899 

945.3 

292.6 

248.9 

93.92%  No 


94.79%  No 


96.00%  No 


95.52%  Yes 


176.69%  Yes 


95.03%  No 


100.00%  No 


95.99%  No 


91 .15%  No 


100.00%  No 


96.67%  No 


97.35%  No 


94.56%  No 


95.29%  No 


99.54%  No 


93.73%  No 
96.95%  No 


99.02%  No 


114.1 2%  Yes 


85.64%  No 


94.96%  No 


80.1 8%  No 


79.64%  No 


11 3.23%  Yes 


131 .93%  Yes 


101 .33%  Yes 


141 .66%  Yes 


157.91%  Yes 


199.75%  Yes 


99.26%  Yes 


86.66%  No 


99.40%  No 


98.22%  No 


120.73%  Yes 


95.67%  No 


125.1 7%  Yes 


124.45%  Yes 


94.25%  No 


97.67%  Yes 


79.73%  No 


77.1 9%  Yes 


132.86%  Yes 


97.51%  No 


95.12%  No 


11 7.01%  Yes 


90.90%  No 


96.95%  No 


Yes 

No 


11 7.56%  Yes 


FP 


CP  DEV 


FP  Prod 


FP  Prod 


FP  Prod 


FP  Prod 


FP  Prod 


CP  DEV 


FP  Prod 


CP  DEV 


FP  Prod 


FP  Prod 


FP  Prod 


FP  Prod 


FP  Prod 


Prod 

Prod 


CP  Prod 


FP  Prod 


FP  Prod 


FP  Prod 


FP 


CP  Prod 


Mix  Prod 


FP  Prod 


FP  Prod 


CP  DEV 


FP  Prod 


FP  DEV 


FP  Prod 


FP 


FP  Prod 


CP  Prod 


FP  Prod 


FP  Prod 


CP  DEV 


FP  DEV 


CP  DEV 


FP  Prod 


FP  Prod 


FP  Prod 


FP  Prod 


FP  Prod 


FP  Prod 


FP  Prod 


CP  Prod 


FP  Prod 


Prod 

Prod 


FP  Prod 


105 


1 1/25/94 


1 1/25/94 


1 1/25/94 


1 2/25/94 


1 2/25/94 


1 2/25/94 


1 2/25/94 


1 2/25/94 


1 2/25/94 


1/25/95 


2/25/95 


2/25/95 


3/25/95 


3/25/95 


3/25/95 


ESI 


4/25/95 


5/25/95 


6/25/95 


6/25/95 


7/25/95 


7/25/95 


8/25/95 


8/25/95 


8/25/95 


8/25/95 


9/25/95 


9/25/95 


9/25/95 


9/25/95 


1 0/25/95 


1 0/25/95 


1 0/25/95 


10/25/95 


11/25/95 


11/25/95 


11/25/95 


1 2/25/95 


1/25/96 


2/25/96 


3/25/96 


4/25/96 


4/25/96 


4/25/96 


4/25/96 


5/25/96 


4.48% 


-0.44% 


-5.52% 


2.88% 


5.02% 


16.01% 


10.70% 


7.75% 


14.73% 


0.49% 


-3.20% 


5.39% 


3.46% 


1.31% 


11.47% 


-15.90% 

3.53%" 


8.23% 


0.96% 


37.94% 


40.70% 


2.21% 


0.29% 


10.63% 


19.16% 


10.66% 


2.07% 


3.74% 


-0.35% 


2.43% 


1.09% 


30.28% 


10.28% 


3.18% 


72.6 


2972.4 


26.1 


7.5 


12.1 


-11.5 


52.1 


33 


0.7 


6/25/96  35.5 


-1.82% 


46.60% 


57.36% 


31.11% 


12.93% 


10.82% 


-1.80% 


8.86% 


35.68% 


0.17% 


-7.10% 

4.39% 


3.28% 


468.7 


224.4 


441 .7 


239.6 


764 


638.5 


165.5 


1478.5 


993.6 


1067.7 


75.7 


816 


50.9 


93.1 


500.4 


16.4 
167.  f 


293.1 


282.5 


98.9 


672.7 


1400.9 


102.2 


122.8 


70.9 


183.8 


93.7 


49.9 


28.7 


345.4 


250.6 


490.9 


312.3 


22.7 


77.7 


110.6 


475.2 


140.3 


228.4 


8154.7 


110 


65.5 


123.9 


628.5 


640.1 


125.5 


418.5 


1118.6 


448.6 


225.4 


467.5 


232.9 


727.5 


550.4 


149.5 


1372.1 


866 


1062.5 


78.2 


774.3 


49.2 


91.9 


448.9 


19.5 

161.4 


270.8 


279.8 


71.7 


478.1 


1370.6 


101.9 


111 


59.5 


166.1 


91.8 


48.1 


28.8 


337.2 


247.9 


376.8 


283.2 


22 


85.6 


115 


511.2 


142.9 


155.8 


5182.3 


83.9 


58 


111.8 


640 


588 


92.5 


417.8 


1172.7 

967.1 


1083.1 


199.6 


215.1 


491 


238.8 


756.1 


611.6 


180.8 


1447.7 


881 


1095.3 


86.1 


853.8 


44.6 


117.3 


287.8 


19.5 

165.7 


228 


285.2 


72.8 


485.5 


1406.1 


102.7 


65.8 


66.6 


127.7 


93.8 


34.3 


29 


373.1 


264.6 


386.3 


236.4 


24.2 


89.3 


116.1 


526.8 


155.9 


157.8 


5547.9 


224.75% 

Yes 

104.79% 

Yes 

95.21% 

Yes 

97.53% 

No 

96.22% 

No 

89.99% 

No 

82.69% 

No 

94.78% 

No 

98.30% 

No 

97.01% 

No 

90.82% 

No 

90.69% 

No 

110.31% 

Yes 

78.35% 

No 

155.98% 

Yes 

100.00% 

09 

97.40% 

118.77% 

Yes 

98.11% 

No 

98.49% 

No 

98.48% 

No 

97.48% 

No 

99.22% 

No 

168.69% 

Yes 

89.34% 

No 

130.07% 

Yes 

97.87% 

No 

140.23% 

Yes 

99.31% 

No 

90.38% 

No 

93.69% 

No 

97.54% 

No 

119.80% 

Yes 

90.91% 

No 

95.86% 

No 

99.05% 

No 

97.04% 

No 

91 .66% 

No 

98.73% 

No 

93.41% 

No 

94.27% 

No 

100.69% 

Yes 

94.03% 

No 

97.04% 

No 

93.04% 

No 

96.56% 

No 
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